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Ranking  Minority  Member 
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As  you  requested,  this  report  assesses  how  best  practices  offer  improvements  to  the  way  the 
Department  of  Defense  (DOD)  incoiporates  new  technology  into  weapon  system  programs.  It  also 
assesses  the  factors  that  can  make  it  difficult  to  mature  technologies  before  they  are  included  on 
weapon  system  programs  and  what  can  be  done  about  them.  We  make  recommendations  to  the 
Secretary  of  Defense  on  how  advanced  technologies  can  be  better  managed  so  they  pose  less  risk 
when  they  are  included  in  weapon  system  designs. 

We  are  sending  copies  of  this  report  to  the  Honorable  William  S.  Cohen,  Secretary  of  Defense;  the 
Honorable  Louis  Caldera,  Secretary  of  the  Army;  the  Honorable  Richard  Danzig,  Secretary  of  the 
Navy;  the  Honorable  F.  Whitten  Peters,  Acting  Secretary  of  the  Air  Force;  the  Honorable  Jacob  J.  Lew, 
Director,  Office  of  Management  and  Budget;  and  to  interested  congressional  committees.  We  will  also 
make  copies  available  to  others  upon  request. 

If  you  have  any  questions  regarding  this  report,  please  call  me  at  (202)  5124841.  Other  key  contacts 
are  listed  in  appendix  III. 


Katherine  V.  Schinasi 
Associate  Director 
Defense  Acquisitions  Issues 
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Purpose 


The  Department  of  Defense  (DOD)  plans  to  increase  its  investment  in  new 
weapons  to  about  $60  billion  in  fiscal  year  2001— a  40-percent  increase 
over  fiscal  year  1997.  DOD  has  high  expectations  from  this  investment:  that 
new  weapons  will  be  better  and  less  expensive  than  their  predecessors  and 
will  be  developed  in  half  the  time.  With  its  traditional  management 
approach— which  has  produced  superior  weapons,  but  at  much  greater 
cost  and  time  than  planned— DOD  will  not  meet  these  expectations. 
Leading  commercial  firms  have  changed  their  practices  for  developing 
products  and  have  achieved  the  kinds  of  results  DOD  seeks.  Maturing  new 
technology  before  it  is  included  in  products  is  one  of  the  main  determinants 
of  these  firms’  successes.  This  practice  holds  promise  for  DOD,  for 
immature  technologies  have  been  a  main  source  of  problems  on  weapon 
systems.  In  response  to  a  request  from  the  Chairman  and  the  Ranking 
Minority  Member,  Subcommittee  on  Readiness  and  Management  Support, 
Senate  Committee  on  Armed  Services,  GAO  assessed  (1)  the  impact  of 
technology  maturity  on  product  outcomes,  (2)  best  practices  for  managing 
new  technologies  and  incorporating  them  into  products,  and  (3)  ways  DOD 
can  adapt  these  practices  to  get  better  outcomes  on  weapon  system 
programs. 


Background 


GAO  reviewed  commercial  and  DOD  experiences  in  incorporating 
23  different  technologies  into  new  product  and  weapon  system  designs. 
The  technologies  were  drawn  from  (1)  six  commercial  firms  recognized  for 
their  success  in  developing  technically  advanced  products  more  quickly 
than  the  products’  predecessors  and  (2)  five  DOD  weapon  system 
programs  that  incorporated  advanced  technologies,  including  some  that 
did  not  encounter  problems  and  some  that  did.  GAO  asked  the  managers  of 
these  technologies  to  assess  the  maturity  of  the  technologies  at  the  point 
they  were  included  in  product  development  by  applying  a  tool,  referred  to 
as  technology  readiness  levels  (TRLs).  The  National  Aeronautics  and  Space 
Administration  and  the  Air  Force  Research  Laboratory  use  TRLs  to 
determine  the  readiness  of  technologies  to  be  incorporated  into  a  weapon 
or  another  type  of  system.  Readiness  levels  are  measured  along  a  scale  of 
one  to  nine,  starting  with  paper  studies  of  the  basic  concept,  proceeding 
with  laboratory  demonstrations,  and  ending  with  a  technology  that  has 
proven  itself  on  the  intended  product. 

The  Air  Force  Research  Laboratory  considers  TRL  6  an  acceptable  risk  for 
a  weapon  system  entering  the  program  definition  stage,  the  point  at  which 
DOD  launches  its  weapon  programs,  and  TRL  7  an  acceptable  risk  for  the 
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engineering  and  manufacturing  development  stage.  This  is  an  important 
distinction  because  leading  commercial  firms  launch  a  new  product  later 
than  DOD,  after  technology  development  is  complete.  They  refer  to  this 
point  as  the  beginning  of  product  development,  the  point  at  which  they 
commit  to  developing  and  manufacturing  the  product.  Typically, 
technology  is  still  being  developed  when  weapon  system  programs  are 
launched;  the  point  at  which  a  weapon  system  is  far  enough  along  to 
compare  to  a  commercial  product  development  is  likely  to  be  at  or  after  the 
start  of  engineering  and  manufacturing  development. 


Results  in  Brief  The  experiences  of  DOD  and  commercial  technology  development  cases 

GAO  reviewed  indicate  that  demonstrating  a  high  level  of  maturity  before 
new  technologies  are  incorporated  into  product  development  programs 
puts  those  programs  in  a  better  position  to  succeed.  The  TRLs,  as  applied 
to  the  23  technologies,  reconciled  the  different  maturity  levels  with 
subsequent  product  development  experiences.  They  also  revealed  when 
gaps  occurred  between  a  technology’s  maturity  and  the  intended  product’s 
requirements.  For  technologies  that  were  successfully  incorporated  into  a 
product,  the  gap  was  recognized  and  closed  before  product  development 
began,  improving  the  chances  for  successful  cost  and  schedule  outcomes. 
The  closing  of  the  gap  was  a  managed  result.  It  is  a  rare  program  that  can 
proceed  with  a  gap  between  product  requirements  and  the  maturity  of  key 
technologies  and  still  be  delivered  on  time  and  within  costs. 

Two  conditions  were  critical  to  closing  the  maturity  gap.  First,  the  right 
environment  for  maturing  technologies  existed.  Key  to  this  environment 
was  making  a  science  and  technology  organization,  rather  than  the 
program  or  product  development  manager,  responsible  for  maturing 
technologies  to  a  high  TRL.  When  a  maturity  gap  persisted,  managers  were 
given  the  flexibility  to  take  the  time  to  mature  the  technology  or  decrease 
product  requirements  so  that  they  could  use  another,  already  mature 
technology.  Second,  both  technology  and  product  managers  were 
supported  with  the  disciplined  processes,  readily  available  information, 
readiness  standards,  and  authority  to  ensure  technology  was  ready  for 
products.  This  support  enabled  these  managers  to  safeguard  product 
development  from  undue  technology  risks.  On  the  other  hand,  immature 
technologies  were  sometimes  incorporated  into  products  for  reasons  such 
as  inflexible  performance  requirements,  increasing  the  likelihood  of  cost 
overruns  and  delays  in  product  development.  Product  managers  had  little 
choice  but  to  accept  the  technologies  and  hope  that  they  would  mature 
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successfully.  However,  the  pressures  of  product  development  made  for  an 
environment  less  conducive  to  maturing  technology. 

For  several  reasons,  DOD  is  likely  to  move  technologies  to  product 
development  programs  before  they  are  mature.  Science  and  technology 
organizations,  which  traditionally  operate  within  fixed  budget  levels,  do 
not  necessarily  have  the  funds  to  mature  technology  to  the  higher  TRLs. 
Programs  are  more  able  to  command  the  large  budgets  necessary  for 
reaching  these  levels.  The  pressures  exerted  on  new  programs  to  offer 
unique  performance  at  low  cost  encourage  acceptance  of  unproven 
technologies.  The  technologies  GAO  reviewed  indicate  these  conditions 
can  be  overcome  on  individual  cases.  DOD  has  several  initiatives 
underway,  such  as  advanced  technology  demonstrations,  that  could  make 
it  more  feasible  for  science  and  technology  organizations  to  mature 
technology  before  it  is  moved  to  product  development  programs.  The 
challenge  will  be  whether  the  lessons  learned  from  these  cases  and 
initiatives  offer  an  approach  that  has  a  DOD-wide  application. 

GAO  makes  recommendations  to  the  Secretary  of  Defense  on  ways  to 
pursue  advanced  technologies  while  lessening  their  potential  for  causing 
problems  on  weapon  system  programs. 


Principal  Findings 


Maturity  of  Technology  at 
Program  Start  Is  an 
Important  Determinant  of 
Success 


The  23  technologies  GAO  reviewed  spanned  a  wide  range  of  readiness 
levels— from  a  low  of  TRL  2  to  a  high  of  TRL  9— when  they  were  included 
in  product  development  programs.  Programs  with  key  technologies  at 
readiness  levels  6  to  8  at  the  time  of  program  launch  met  or  were  meeting 
cost,  schedule,  and  performance  requirements.  All  of  the  commercial 
technologies  and  a  few  of  the  DOD  technologies  fell  into  this  category.  For 
example,  Ford  managed  its  voice-activated  control  technology  to  TRL  8— a 
10-year  effort— before  introducing  it  on  the  1999  Jaguar.  Similarly,  the 
Defense  Advanced  Research  Projects  Agency  matured  a  revolutionary 
periscope  technology  to  TRL  9  before  it  was  included  on  the  Virginia  class 
attack  submarine.  DOD  programs  that  accepted  technologies  at  a  readiness 
level  of  5  or  less  experienced  significant  cost  and  schedule  increases  due, 
in  part,  to  problems  with  the  technologies.  DOD’s  acceptance  of 
technologies  at  level  4  or  lower  was  not  unusual.  For  example,  the  key 
technologies  for  the  Army’s  brilliant  antiarmor  submunition  were  at  levels  2 
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and  3  when  weapon  system  development  began.  At  these  levels,  DOD  had  a 
significant  gap  in  technology  maturity  at  the  start  of  the  program.  The  gap 
was  not  closed  until  well  into  the  development  program,  and  problems  with 
the  technologies  were  a  main  contributor  to  the  program’s  88-percent  cost 
growth  and  62-percent  slip  in  schedule. 


Controllable  Conditions 
Affect  How  Well  a 
Technology’s  Inclusion  on  a 
Product  Can  Be  Managed 


Closing  the  technology  development  gap  before  beginning  product 
development  was  the  result  of  good  technology  maturation  practices  and 
sound  methods  for  moving  technologies  to  products.  The  more  successful 
of  the  23  technologies  were  managed  by  science  and  technology 
organizations  until  they  reached  at  least  TRL  6  and  more,  often  TRL  8  or 
higher.  This  environment  was  an  important  condition  for  successfully 
maturing  technologies,  as  it  allowed  room  for  unexpected  results  such  as 
test  “failures,”  which  are  considered  normal  events  in  developing 
technologies.  To  match  technology  maturity  and  product  requirements, 
managers  also  had  the  option  of  waiting  until  technologies  matured  or 
changing  product  requirements  so  that  an  already  mature  technology  could 
be  used.  For  example,  Hughes  deferred  the  development  of  the  HS-702 
satellite  until  critical  solar  cell  technology  had  matured — a  process  that 
took  over  10  years.  Also,  Navy  managers  accepted  an  existing  weapon 
ejection  system  on  the  Virginia  class  attack  submarine  when  technology 
failed  to  mature  as  expected.  In  contrast,  performance  requirements  for  the 
Comanche  helicopter  were  inflexible;  requirements  mandated  the  inclusion 
of  advanced  sensors  and  avionics  technologies,  despite  their  immaturity. 
The  Comanche  program  has  experienced  cost  growth  and  schedule  delays, 
partly  attributable  to  the  inclusion  of  these  technologies. 


In  the  more  successful  cases,  technology  and  product  managers  were  given 
the  authority  and  tools  to  move  technology  only  when  it  was  at  high 
readiness  levels.  Disciplined  processes  provided  managers  credible 
information  on  the  status  of  technologies  and  high  standards  for  assessing 
readiness.  Science  and  technology  managers  developed  technologies  to 
standards  acceptable  to  product  managers  who  could  reject  those 
technologies  that  fell  short.  For  example,  Ford’s  science  and  technology 
managers  use  agreed-upon  standards  forjudging  technology  readiness,  and 
all  new  technologies  follow  the  same  maturation  process.  Ford’s  product 
managers  are  also  empowered  to  say  no  when  technologies  are  not  deemed 
mature.  Recently,  the  Jaguar  vehicle  team  rejected  night  vision  technology 
at  TRL  8  because  it  did  not  meet  cost  objectives.  DOD  program  managers 
that  had  to  accept  immature  technologies  had  less  information  available  to 
guide  them.  For  example,  key  technologies  for  the  brilliant  antiarmor 
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submunition  program  bypassed  Army  science  and  technology 
organizations,  forcing  the  program  manager  to  accept  the  technologies 
with  little  information  about  their  readiness.  Often,  the  tools  used  to  assess 
the  technologies’  status  failed  to  identify  high  risks;  the  TRLs  indicate  that 
risks  on  the  problematic  technologies  were  often  high.  Also,  the  greater 
pressures  to  meet  cost  and  schedule  goals  in  product  development 
provided  a  less  forgiving  environment  for  fledgling  technologies. 


Impediments  to  Adopting 
Best  Practices  for 
Technology  Inclusion  in 
DOD  Are  Surmountable 


Leading  commercial  firms  have  put  the  organizations,  tools,  and  other 
practices  in  place  to  foster  technology  development  and  improve  the 
outcomes  of  product  developments  as  a  matter  of  necessity.  The  large 
investment  required  for  a  new  product  and  the  risks  to  that  investment  if 
the  product  does  not  meet  customer  needs  reinforce  these  practices.  The 
DOD  cases  that  followed  a  similar  approach — the  Advanced  Amphibious 
Assault  Vehicle  and  the  Virginia  class  attack  submarine — have  so  far 
avoided  problems  with  key  technologies.  Yet  these  cases  are  not  the  norm 
for  DOD  programs.  DOD  programs  operate  under  conditions  that  make  it 
more  difficult — and  less  rewarding — to  separate  technology  from  product 
development  and  to  allow  technology  to  reach  high  maturity  before  being 
included  in  an  acquisition  program. 

It  is  easier  for  weapon  system  programs  to  fund  technology  development  at 
higher  readiness  levels  because  they  attract  much  bigger  budgets  than 
science  and  technology  projects.  DOD  typically  does  not  fund  science  and 
technology  organizations  to  take  technology  past  the  feasibility  stage — 
TRL  5.  As  a  practical  matter,  it  is  often  necessary  to  move  immature 
technology  to  a  weapon  system  program  to  get  needed  funds  and 
management  support.  New  programs  are  pressured  to  include  immature 
technologies  that  offer  significant  performance  gains.  These  pressures 
come  from  the  user’s  perception  of  the  threat,  technologists  that  see  the 
program  as  an  opportunity  to  apply  a  new  technology,  and  funding 
competition  that  rewards  weapon  systems  with  unique  features. 

DOD  and  the  services  have  several  initiatives  for  improving  the  technology 
development  process  and  reducing  weapon  system  cycle  times.  These 
include  defense  technology  objectives,  advanced  technology 
demonstrations,  advanced  concept  technology  demonstrations,  the  Army’s 
new  scout/cavalry  vehicle,  and  the  Air  Force’s  Integrated  High 
Performance  Turbine  Engine  Technology  Program.  These  initiatives  are 
aimed  at  putting  the  science  and  technology  organizations  and  funding  in 
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place  to  bring  technologies  to  higher  readiness  levels  before  they  are 
included  in  weapon  system  programs. 


Recommendations  GAO  recommends  that  the  Secretary  of  Defense  adopt  a  disciplined  and 

knowledge-based  approach  of  assessing  technology  maturity,  such  as 
TRLs,  DOD-wide,  and  establish  the  point  at  which  a  match  is  achieved 
between  key  technologies  and  weapon  system  requirements  as  the  proper 
point  for  committing  to  the  development  and  production  of  a  weapon 
system.  GAO  also  recommends  that  the  Secretary  (1)  require  that 
technologies  needed  to  meet  a  weapon’s  requirements  reach  a  high 
readiness  level  (analogous  to  TRL  7)  before  making  that  commitment, 

(2)  extract  lessons  from  successful  technology  inclusion  cases  for 
application  to  future  technology  inclusion  efforts,  and  (3)  empower 
program  managers  to  refuse  to  accept  key  technologies  with  low  levels  of 
maturity  by  making  decisions  on  individual  programs  that  reinforce  a  best 
practice  approach  to  technology  maturation  and  inclusion.  These 
recommendations  appear  in  full  in  chapter  5. 


AgeilCy  Comments  DOD  generally  agreed  with  the  report  and  its  recommendations.  A  detailed 

"  '  “  discussion  of  DOD’s  comments  appear  in  appendix  I. 
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Introduction 


A  central  piece  of  the  National  Military  Strategy  is  the  military  capability 
represented  by  advanced  weaponry.  The  Department  of  Defense  (DOD) 
plans  to  increase  its  annual  investment  in  new  weapons  to  about  $60  billion 
by  fiscal  year  2001 — a  40-percent  increase  over  fiscal  year  1997.  DOD  has 
high  expectations  from  this  investment:  that  new  weapons  will  be  better 
and  less  expensive  than  their  predecessors  and  will  be  developed  in  half 
the  time.  These  expectations  frame  a  great  challenge  for  managers  of 
programs.  The  traditional  management  approach — which  has  produced 
superior  weapons  but  at  much  greater  cost  and  time  than  planned — will  not 
meet  these  expectations.  Cycle  times — the  time  to  develop  a  new 
weapon — can  be  so  long  that  the  technology  a  weapon  is  designed  with 
becomes  obsolete  before  it  can  be  produced.  Costs  of  new  weapons  have 
reached  the  point  that  significantly  fewer  can  be  bought  than  planned. 
These  are  not  new  issues,  but  they  have  become  more  pressing  as  the  pace 
and  sophistication  of  foreign  and  commercial  technology  have  increased, 
complicating  a  national  security  environment  of  unknown  threats. 

Leading  commercial  firms  have  changed  the  way  they  develop  products 
and  have  achieved  the  kinds  of  results  DOD  seeks,  often  yielding  more 
sophisticated  products  in  half  the  time  formerly  needed.  Industry  experts 
estimate  that  resolving  technology  problems  before  product  development 
begins  results  in  10  times  the  savings  compared  to  correcting  problems 
afterward.  In  this  sense,  technology  maturity  breeds  product  success.  The 
practices  leading  firms  use  to  mature  and  transition  technology  to  products 
hold  promise  for  DOD,  for  immature  technologies  have  been  main  sources 
of  problems  on  weapon  systems.  We  have  previously  reported  on  the 
different  elements  of  knowledge  firms  insist  on  to  get  better  products  to 
market  faster.  Of  these,  no  element  is  more  important  than  having 
technology,  advanced  enough  to  meet  requirements  but  also  mature 
enough  to  be  predictably  managed,  available  at  the  start  of  the  product 
development  cycle.  Maturing  new  technology  before  it  is  included  on  a 
product  is  perhaps  the  most  important  determinant  of  the  success  of  the 
eventual  product — or  weapon  system.  It  is  the  topic  of  this  report. 


Separating  Technology 
Development  From 
Product  Development 
Is  a  Best  Practice 


The  cycle  for  placing  better  capabilities  in  the  hands  of  users — both 
military  and  commercial — can  be  described  as  consisting  of  technology 
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development,  product  development,  and  production.  In  a  1998  report,1  we 
characterized  the  knowledge  needed  on  a  new  product  as  consisting  of 
three  knowledge  points:  when  a  match  is  made  between  a  customer’s 
requirements  and  the  available  technology;  when  the  product’s  design  is 
determined  to  be  capable  of  meeting  performance  requirements;  and  when 
the  product  is  determined  to  be  producible  within  cost,  schedule,  and 
quality  targets  (see  fig.  1.1).  We  found  that  this  knowledge,  when  obtained 
at  the  right  time  and  in  the  right  sequence — technology,  design,  and 
manufacturing — was  a  best  practice.  This  practice  lowered  product 
development  risks,  reduced  cycle  times  and  costs,  and  resulted  in 
smoother  production  programs. 


Figure  1.1:  Cycle  for  Providing  Users  a  Product  With  Better  Capabilities 
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Leading  commercial  firms  recognize  a  distinct  difference  between 
technology  development  and  product  development;  accordingly,  they 
develop  technology  before  introducing  it  into  product  development 
programs.  They  minimize  risk,  improve  cost  and  schedule  outcomes, 
reduce  cycle  time,  and  improve  quality  during  product  development  by 


‘Best. Practices:  Successful  Application  to  Weapon  Acquisitions  Requires  Changes  in  POP’s 

Environment  (GA0/NSIAD-98-56,  Feb.  24, 1998). 


Page  13 


GAO/NSIAD-99-162  Best  Practices 


Chapter  1 
Introduction 


gaining  significant  knowledge  about  a  technology  before  launching  the 
product  development.  Scientists  and  technologists — different  people  than 
those  that  manage  product  developments — manage  the  development  of 
technology  until  it  is  ready  to  be  included  in  the  design  of  a  product. 

Program  launch  is  the  point  at  which  a  firm  defines  a  product’s 
performance,  cost,  and  schedule  estimates  and  begins  making  a  large 
investment  in  human  capital,  facilities,  and  materials — an  investment  that 
increases  continuously  as  the  product  approaches  the  point  of 
manufacture.  It  includes  a  commitment  to  manufacture  the  product. 
Therefore,  program  launch  and  the  start  of  product  development  are 
synonymous  within  commercial  firms.  Protecting  this  investment  provides 
a  strong  incentive  for  firms  to  minimize  the  potential  for  technology 
development  problems  during  the  product  phase  and  cause  delays. 
Confining  delays  in  maturing  technology  to  a  time  prior  to  launch — in  an 
environment  where  small  teams  of  technologists  work  in  laboratories  and 
are  dedicated  to  perfecting  the  technology — is  critical  to  saving  time  and 
money.  If  delays  occur  during  product  development,  when  a  large 
engineering  force  is  in  place  to  design  and  manufacture  the  product,  they 
would  be  much  more  costly.  In  fact,  industry  experts  estimate  that 
identifying  and  resolving  a  problem  before  product  development  can  reap  a 
10-fold  savings  compared  to  correcting  the  problem  after  launch  and  that 
correcting  the  same  problem  in  the  manufacturing  stage  would  be  even 
more  costly. 

Leading  commercial  firms  have  found  that  managing  technology 
development  separately  from  and  before  product  development  is  a  major 
reason  they  have  been  able  to  reduce  product  cycle  times.  As  a  whole, 

50  to  70  percent  reductions  in  cycle  times  are  not  unrealistic  achievements 
by  leading  commercial  firms.  For  instance,  leaders  in  the  automobile 
industry  have  reduced  cycle  times  from  7  years  to  2  years,  or  by  about 
70  percent.  The  consumer  electronics  industry  has  recently  reduced  its 
cycle  time  from  2  years  to  6  months,  and  the  commercial  aircraft  industry 
has  achieved  reductions  of  50  percent.  Leading  commercial  firms  have 
found  that  reducing  the  product  development  cycle  time  brings  products  to 
market  faster,  results  in  an  increased  market  share,  and  helps  to  keep 
products  from  becoming  technologically  obsolete. 
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Technology  and 
Product  Development 
Conducted  at  the  Same 
Time  Within  DOD 


DOD’s  process  for  developing  and  manufacturing  weapon  systems  is 
described  as  a  cycle  consisting  of  phases.  These  phases  are  concept 
exploration,  program  definition  and  risk  reduction,  engineering  and 
manufacturing  development,  and  production  and  fielding.  The  basic 
process  of  gathering  knowledge  about  technology,  design,  and 
manufacturing  is  followed,  but  in  practice,  the  DOD  cycle  does  not  make  a 
clear  distinction  between  technology  development  and  product 
development.  The  launch  of  a  program  in  DOD  usually  takes  place  several 
years  before  the  beginning  of  product  development  does  in  leading 
commercial  firms. 


In  fact,  a  new  weapon  system  program  is  normally  launched  at  the  start  of 
the  program  definition  and  risk  reduction  phase,  which  is  often  in  the  midst 
of  technology  development,  while  most  product  development  activities  do 
not  begin  until  the  engineering  and  manufacturing  development  phase. 
Consequently,  technology,  design,  and  manufacturing  knowledge  is 
attained  concurrently-in  the  higher  cost  environment  that  characterizes 
product  development-throughout  the  weapon  system  phases.  In  our 
February  1998  report,  we  noted  that  such  technology  development 
problems  are  a  major  cause  of  cost  increases  and  schedule  delays  on  DOD 
weapon  system  programs.  The  phases  in  DOD’s  weapon  system  acquisition 
cycle  and  the  knowledge  gathering  process,  as  it  is  typically  followed,  are 
shown  in  figure  1.2. 
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Figure  1.2:  DOD’s  Weapon  System  Acquisition  Cycle 
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DOD’s  process  also  has  organizational  and  budgetary  implications. 
Activities  accomplished  in  the  first  three  phases  of  the  acquisition  cycle  use 
research  and  development  funds  whereas  production  programs  use 
procurement  funds.  Generally,  DOD’s  science  and  technology  (S&T) 
community  is  responsible  for  basic  research,  applied  research,  and 
advanced  technology  development  to  produce  generic,  rather  than 
weapon-specific,  technologies.  Its  goal  is  to  conduct  research,  develop 
technology,  and  farm  these  efforts  for  potential  military  application,  such 
as  a  weapon  system.  The  S&T  community  also  uses  research  and 
development  funds,  but  its  work  generally  precedes  the  acquisition  cycle. 
Weapon  system  program  managers,  who  receive  most  of  DOD’s  research 
and  development  budget,  apply  generic  technologies  to  specific  weapon 
systems.  However,  they  often  become  responsible  for  completing 
development  of  generic  technologies  as  well.  The  allocation  of  DOD’s  fiscal 
year  1999  research  and  development  funds  to  these  categories  is  shown  in 
figure  1.3. 
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Figure  1.3:  Allocation  of  DOD’s  Fiscal  Year  1999  Research  and  Development  Funds 


Source:  DOD 

S&T  officials  stated  their  role  is  to  show  that  technology  is  feasible  through 
laboratory  experiments  or  demonstrations.  It  is  often  at  this  point  that  the 
technology’s  military  potential  will  be  identified  and  the  technology  will  be 
harvested  for  inclusion  on  a  weapon  system.  Because  the  technology  is  still 
not  mature,  its  development  will  be  completed  as  part  of  the  weapon 
system’s  design  and  development,  under  the  authority  of  the  weapon 
system  manager  and  apart  from  the  S&T  community. 


Shorter  Acquisition 
Cycle  Times  Are 
Needed 


DOD’s  weapon  acquisition  cycle  times  average  between  10  to  15  years — far 
longer  than  the  cycle  time  for  commercial  products.  To  an  extent,  DOD’s 
cycle  times  are  longer  because  they  start  earlier  than  commercial  cycles 
and  often  entail  more  complex  products.  Compounding  the  length  of  the 
weapon  system  development  cycle  is  its  unpredictability.  Over  the  years, 
we  have  issued  numerous  reports  highlighting  cost  overruns  and  schedule 
delays  during  the  product  development  cycle,  for  which  technology 
development  problems  were  a  major  cause.  These  problems  require 
additional  technology  development  activities  to  take  place  at  a  time  when 
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the  product  should  be  undergoing  design  and  manufacturing  development. 
As  a  result,  the  pace  of  technology  advances  outruns  the  time  to  develop  a 
weapon  system  and  some  of  the  more  mature  components  designed  into  a 
weapon  system  become  obsolete  before  the  weapon  is  manufactured.  For 
example,  the  F-22  will  have  almost  600  obsolete  components  by  fiscal  year 
2000  while  the  aircraft  is  still  in  development. 

The  longer  a  weapon  system’s  development  cycle,  the  more  prone  the 
program  is  to  management  and  funding  changes.  According  to  DOD,  an 
11-year  development  program  historically  encounters  a  30-percent  cost 
growth  over  time.  Based  on  historical  averages,  DOD  calculates  that  the 
typical  program  will  have  four  different  program  managers,  eight  defense 
acquisition  executives,  and  seven  Secretaries  of  Defense — all  of  who  are 
major  influences  and  decisionmakers  on  the  program.  In  addition,  the 
program  will  have  gone  through  11  annual  budget  cycles  in  which  funding 
changes  could  have  occurred  and  affected  the  program’s  content. 

The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  has  stated 
that  cycle  time  reduction  is  necessary  to  meet  DOD’s  goals  of  delivering 
emerging  technologies  to  warfighters  in  less  time  and  at  lower  costs.  The 
Under  Secretary  has  set  a  goal  to  reduce  the  average  acquisition  cycle  time 
for  all  program  starts  in  fiscal  year  1999  and  beyond  by  60  percent  over 
historical  averages.  Reductions  in  cycle  times  will  (1)  allow  for  earlier 
fielding  of  increased  capabilities,  (2)  reduce  costs,  (3)  free  up  funds  for 
more  programs,  (4)  reduce  the  potential  for  components  becoming 
obsolete,  and  (5)  take  more  frequent  advantage  of  technology  advances 
found  in  the  commercial  world.  An  emphasis  on  shorter  cycle  times  may 
also  reduce  the  tendency  to  add  technological  advances  that  are  unproven 
and  immature  into  weapon  acquisition  programs.  To  help  achieve  this  goal, 
DOD  is  working  on  several  efforts  such  as  Defense  Acquisition  Pilot 
Programs,  the  Defense  Reform  Initiatives,  and  many  acquisition  reform 
projects.  The  Under  Secretary  has  also  advocated  adopting  the  practices  of 
leading  commercial  firms  and  taking  a  more  evolutionary  approach  to 
developing  weapon  systems,  which  would  lessen  the  amount  of  technology 
development  initially  attempted  within  a  weapon  system  program. 


Objectives,  Scope,  and 
Methodology 


The  Chairman  and  the  Ranking  Minority  Member,  Subcommittee  on 
Readiness  and  Management  Support,  Senate  Committee  on  Armed 
Services,  requested  that  we  examine  various  aspects  of  the  acquisition 
process  to  determine  whether  the  application  of  best  practices  can  improve 
program  outcomes.  To  date,  we  have  issued  reports  on  advanced  quality 
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concepts,  earned  value  management,  management  of  a  product  from 
development  to  production,  and  management  of  key  suppliers  (see  related 
GAO  products).  This  report  covers  the  inclusion  of  technology  into  weapon 
system  programs,  and  is,  in  a  sense,  a  prequel  to  our  report  on  product 
development.  Our  overall  objective  was  to  determine  whether  best 
practices  offer  methods  to  improve  the  way  DOD  matures  new  technology 
so  that  it  can  be  assimilated  into  weapon  system  programs  with  less 
disruption.  Specifically,  we  assessed  (1)  the  impact  of  technology  maturity 
on  product  outcomes,  (2)  best  practices  for  managing  new  technologies 
and  incorporating  them  into  products,  and  (3)  ways  DOD  can  adapt  best 
practices  to  achieve  better  outcomes  on  weapon  system  programs. 

Our  methodology  consisted  of  analyzing  23  commercial  and  DOD 
technologies  that  had  transitioned  or  attempted  to  transition  into  product 
development  programs.  The  technologies  were  drawn  from  six  commercial 
firms  recognized  for  their  success  in  developing  technically  advanced 
products  more  quickly  than  their  predecessors  and  five  weapon  system 
programs  that  incorporated  advanced  technologies,  including  some  that 
did  not  encounter  problems  and  some  that  did.  We  asked  the  managers  of 
these  technologies  to  apply  a  tool,  referred  to  as  technology  readiness 
levels  (TRLs),  for  our  analysis.  The  managers  used  TRLs  to  judge  the 
maturity  of  the  technologies  at  the  time  they  had  entered  product 
development  or  were  included  in  programs.  The  National  Aeronautics  and 
Space  Administration  (NASA)  originally  developed  TRLs,  and  the  Air  Force 
Research  Laboratory  uses  them  to  determine  when  technologies  are  ready 
to  be  handed  off  from  S&T  managers  to  product  development  managers. 
We  held  discussions  with  the  DOD  and  NASA  users  of  TRLs  to  better 
understand  their  applicability  to  our  review.  They  stated  that  TRLs  can  be 
used  as  general  indicators  of  a  technology’s  readiness  level  and  associated 
risk  of  including  the  technology  into  a  product  development  program,  given 
its  TRL  at  that  time.  TRLs  are  more  fully  explained  in  chapter  2. 

To  understand  the  best  practices  the  commercial  sector  used  to  include 
technologies  in  product  development  programs,  we  conducted  literature 
searches  and  focused  those  searches  as  the  review  progressed.  On  the 
basis  of  the  searches,  we  identified  a  number  of  commercial  firms  with 
innovative  technology  development  processes  for  including  new  or 
advanced  technologies  into  new  products.  We  used  structured  interview 
questions  sent  in  advance  of  our  visits  to  gather  uniform  and  consistent 
information  about  each  firm’s  process  and  practice  and  the  results 
achieved.  In  addition,  we  examined  four  specific  technology  cases — Ford’s 
night  vision,  adaptive  cruise  control,  and  voice  activated  controls  and 
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Hughes’  solar  cell  array — to  better  understand  their  processes  and 
practices.  The  commercial  firms  we  visited  were 

•  Ethicon-Endo  Surgery  (medical  device  manufacturer),  Division  of 
Johnson  and  Johnson,  Cincinnati,  Ohio; 

•  Ford  Motor  Company  (automobile  manufacturer),  Dearborn,  Michigan; 

•  Harris  Semiconductor  (semiconductor  manufacturer),  Melbourne, 
Florida; 

•  Hughes  Space  and  Communications  (satellite  and  spacecraft 
manufacturer),  Los  Angeles,  California; 

•  3M  (commercial  products  manufacturer),  St.  Paul,  Minnesota;  and 

•  Motorola  Corporate  Research  Headquarters  (communications 
technology  manufacturer),  Schaumburg,  Illinois,  and  Motorola  Land 
Mobile  Products  Sector,  Plantation,  Florida. 

We  also  attended  and  participated  in  conferences  and  workshops  with 
recognized  leaders  in  the  acquisition  field  to  obtain  information  on  how 
organizations  are  improving  their  acquisition  processes.  Finally,  we 
interviewed  officials  from  trade  organizations  concerning  the  application 
of  commercial  practices  to  DOD  operations. 

To  better  understand  DOD’s  technology  inclusion  process,  we  selected  19 
advanced  technologies  that  had  been  included  in  5  DOD  weapon  system 
programs  that  were  in  various  stages  of  the  acquisition  process.  We 
collected  technical  reports,  acquisition  management,  and  risk  management 
documentation  about  the  technologies.  In  addition,  we  interviewed  S&T 
and  acquisition  program  management  officials  about  each  technology’s 
development  history,  costs,  and  current  status.  The  technologies  and 
programs  reviewed  were 

•  acoustic  sensor,  infrared  seeker,  inertial  measurement  unit,  tandem 
shaped  charge  warhead,  and  processor  technologies  from  the  Army’s 
Brilliant  Anti-Armor  Submunition  (referred  to  as  BAT)  Program; 

•  rotor,  engine,  integrated  avionics,  forward  looking  infrared,  and  helmet 
mounted  display  technologies  from  the  Army’s  Comanche  Helicopter 
Program; 

•  nonpenetrating  periscope  and  weapon  ejection  system  technologies 
from  the  Navy’s  Virginia  class  attack  submarine  program; 

•  high  speed  planing  craft,  power  dense  diesel  engine,  lightweight 
composite  armor,  high  power  water  jet,  moving  map  and  advanced 
navigation  technologies  from  the  Marines’  Advanced  Amphibious 
Assault  Vehicle  (AAAV)  Program;  and 
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•  laser  and  beam  control  technologies  from  the  Air  Force’s  Airborne  Laser 
(ABL)  Program. 

To  determine  relevant  DOD  policy  and  initiatives,  we  obtained  documents 
and  interviewed  officials  of  the  Office  of  the  Secretary  of  Defense;  the 
Defense  Advanced  Research  Projects  Agency  (DARPA);  and  Army,  Navy 
and  Air  Force  Science  and  Technology  organizations.  We  also  had 
discussions  with  former  DOD  officials  and  industry  experts  about  DOD 
acquisition  policies  and  practices. 

Even  though  we  selected  firms  with  product  lines  of  varying  complexity, 
we  did  not  concentrate  only  on  firms  whose  products  had  the  most  in 
common  with  weapon  systems.  Such  an  approach  would  have  limited  our 
ability  to  include  firms  recognized  as  the  best  at  including  new,  advanced 
technologies  into  programs.  In  our  analysis,  we  concentrated  on  the 
criteria  and  knowledge  used  to  support  technology  readiness  decisions. 
Although  the  approach  from  product  to  product  may  vary,  the  basic 
processes  and  standards  leading  commercial  firms  applied  to  technology 
inclusion  decisions  were  consistent.  We  were  limited,  however,  in  our 
ability  to  obtain  and  present  some  relevant  data  that  commercial 
companies  considered  proprietary  in  nature.  This  information  included 
funding  amounts  for  investing  in  technology  development,  details  on 
technological  innovations,  and  some  specific  data  from  recent  technology 
inclusion  successes.  Our  report  highlights  the  best  commercial  practices 
for  including  technology  into  product  development  programs.  As  such,  they 
are  not  intended  to  describe  all  commercial  industry  practices  or  to  suggest 
that  commercial  firms  do  not  have  any  flaws. 

We  conducted  our  review  between  March  1998  and  June  1999  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Maturity  of  Technology  at  Program  Start  Is  an 
Important  Determinant  of  Success 


The  experiences  of  the  DOD  and  commercial  technology  development 
cases  we  reviewed  indicate  that  demonstrating  a  high  level  of  maturity 
before  allowing  new  technologies  into  product  development  programs  puts 
those  programs  in  a  better  position  to  succeed.  Simply  put,  the  more 
mature  technology  is  at  the  start  of  the  program,  the  more  likely  the 
program  will  succeed  in  meeting  its  objectives.  Technologies  that  were 
included  in  a  product  development  before  they  were  mature  later 
contributed  to  cost  increases  and  schedule  delays  in  those  products. 

We  found  an  analytical  tool— TRLs— that  can  assess  the  maturity  level  of 
technology  as  well  as  the  risk  that  maturity  poses  if  the  technology  is 
included  in  a  product  development.  The  tool  associates  different  TRLs  with 
different  levels  of  demonstrated  performance,  ranging  from  paper  studies 
to  proven  performance  on  the  intended  product.  The  value  of  using  the  tool 
is  that  it  can  presage  the  likely  consequences  of  incorporating  a  technology 
at  a  given  level  of  maturity  into  a  product  development,  enabling 
decisionmakers  to  make  informed  choices.  TRLs  proved  to  be  reliable 
indicators  of  the  relative  maturity  of  the  23  technologies  reviewed,  both 
commercial  and  military,  and  their  eventual  success  after  they  were 
included  in  product  development  programs. 


Technology  Maturity 
Can  Be  Measured  and 
Its  Consequences  for 
Products  Can  Be 
Forecast 


Successful  technologies  progress  from  initial  concept  to  proven 
performance,  whether  they  are  developed  in  the  laboratory  or  in  the 
factory,  by  commercial  industry  or  DOD.  The  Air  Force  Research 
Laboratory  has  adapted  and  uses  TRLs  to  measure  the  key  steps  in  this 
progression  toward  inclusion  into  weapon  systems.  TRLs  are  measured 
along  a  scale  of  one  to  nine,  starting  with  paper  studies  of  the  basic  concept 
and  ending  with  a  technology  that  has  proven  itself  in  actual  usage  on  the 
intended  product.  A  detailed  description  of  TRLs  is  provided  in 
appendix  n,  but  the  following  hypothetical  example  about  an  airborne 
communications  radio  can  illustrate  the  readiness  levels. 


First,  the  idea  for  a  new  radio  is  conceived.  The  idea  reaches  TRL  3  when 
analytical  studies  and  some  tests  of  the  technology’s  elements,  such  as  a 
circuit,  back  it  up.  When  initial  hand-built  versions  of  all  of  the  radio’s  basic 
elements  are  connected  and  tested  together,  the  radio  reaches  TRL  5.  This 
is  sometimes  referred  to  as  a  “breadboard”  article;  although  it  may  function 
like  a  radio,  it  does  not  look  like  one  because  the  individual  parts  are 
attached  to  plywood  and  hand-wired  together.  When  the  technology  is  built 
into  a  generic  model,  which  is  well  beyond  the  breadboard  tested  in  TRL  5, 
and  demonstrated  in  a  laboratory  environment,  the  radio  reaches  TRL  6. 
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This  model  represents  the  last  level  of  demonstration  before  the  radio 
becomes  tailored  for  application  to  a  specific  aircraft.  When  the 
components  are  assembled  inside  a  case  that  resembles  the  final  radio 
design  and  are  demonstrated  aboard  a  surrogate  for  the  intended  aircraft, 
the  radio  reaches  TRL  7.  TRL  8  is  reached  when  the  radio  is  put  in  its  final 
form,  installed  in  the  intended  aircraft’s  cockpit,  and  tested  in  conjunction 
with  the  other  aircraft  equipment  with  which  it  must  interface.  TRL  9  is 
achieved  when  the  radio  is  successfully  operated  on  the  aircraft  through 
several  test  missions.  Unexpected  problems  can  arise  at  every  level,  and 
effort  must  be  expended  to  overcome  them.  This  effort  takes  time  and  can 
delay  the  progress  to  the  next  readiness  level. 

Once  a  technology’s  readiness  level  has  been  established,  the  risks  of 
including  that  technology  in  a  product  development  can  be  assessed. 
Unlike  S&T  projects,  for  which  the  main  objective  is  to  develop  knowledge, 
a  product  development’s  objective  is  to  deliver  products  that  meet  strict 
cost,  schedule,  and  performance  targets.  We  found  that  most  leading 
commercial  firms,  after  they  had  translated  their  own  methods  of  assessing 
risk  into  TRLs,  determined  that  a  TRL  8  was  required  before  they  allowed  a 
new  technology  into  a  product  development.1  DOD  launches  a  program  in 
the  program  definition  and  risk  reduction  phase — much  earlier  than  the 
leading  commercial  firms  do.  According  to  the  Air  Force  Research 
Laboratory,  a  TRL  6  is  required  for  a  technology  to  be  an  acceptable  risk  for 
a  program  in  that  phase.  When  weapon  system  development  reaches  the 
engineering  and  manufacturing  development  phase,  it  more  nearly 
approximates  the  point  at  which  a  commercial  product  development 
program  would  start.  The  Air  Force  Research  Laboratory  depicts  a 
technology  at  TRL  7  as  an  acceptable  risk  for  this  phase — technologies  at 
lower  levels  would  be  considered  high  risks. 

The  lower  the  level  of  technology  readiness,  the  more  ground  must  be 
covered  to  bring  the  technology  to  the  point  at  which  it  can  meet  the 
intended  product’s  cost,  schedule,  and  performance  requirements  with 
little  risk  (see  fig.  2.1). 


‘An  exception  to  this  is  space  systems  technology.  Space-based  technologies  are  generally  included  on  a 
development  program  once  they  have  been  prototyped  and  ground  tested — a  TRL  6,  the  highest  level 
attainable  short  of  space  operation. 
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The  gap  between  the  maturity  of  the  technology  and  the  product’s 
requirements  represents  the  risks  or  unknowns  about  the  technology.  As 
each  succeeding  level  of  readiness  is  demonstrated,  unknowns  are 
replaced  by  knowledge  and  the  gap  becomes  smaller.  Ideally,  the  gap  is 
closed  before  a  new  technology  is  included  in  a  new  product’s  design, 
although  the  Air  Force  Research  Laboratory  accepts  the  amount  of  risk  at 
TRL  7  for  a  program  entering  engineering  and  manufacturing  development. 
Technologies  that  reach  TRL  7  or  higher  at  the  start  of  product 
development  allow  product  managers  to  focus  their  attention  on 
integrating  the  technologies  and  proving  out  the  product  design. 
Technologies  that  are  included  at  lower  maturity  levels  require  more  of  the 
product  managers’  attention  and  resources,  as  basic  knowledge  about 
those  technologies  must  still  be  gained. 

Thus,  a  major  purpose  served  by  TRLs  is  to  reveal  the  gap  between  a 
technology’s  maturity  and  the  maturity  demanded  for  successful  inclusion 
in  the  intended  product.  With  TRLs  as  guides,  the  options  available  to 
decisionmakers  can  be  framed.  Given  that  a  key  determinant  of  achieving 
cost  and  schedule  outcomes  for  a  product  development  is  the  technology’s 
maturity  at  product  launch,  decisionmakers  can  either  (1)  delay  product 
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development  until  the  technology  is  matured  to  a  high  enough  readiness 
level  or  (2)  reduce  the  product’s  requirements  so  that  a  less  advanced,  but 
more  mature,  technology  can  suffice.  If  it  is  perceived  that  the 
requirements  of  the  product  cannot  be  lowered  and  the  product  launch 
cannot  be  delayed  until  the  requisite  technology  is  of  a  sufficient  readiness 
level,  then  the  remaining  option  is  to  launch  the  product  development  with 
the  immature  technology.  If  this  option  is  chosen,  then  the  success  of  the 
product  development  will  depend  heavily  on  the  product  manager’s  ability 
to  simultaneously  close  the  technology  maturity  gap  and  develop  the 
product  for  manufacture,  which  is  a  very  challenging  task. 

TRLs  do  not  represent  strictures  that  must  be  adhered  to  without 
exception.  According  to  the  people  in  DOD  who  have  used  TRLs,  there  are 
occasions  when  a  lower  than  expected  TRL  can  be  accepted,  such  as  when 
the  product  development’s  schedule  and  resources  are  generous  enough 
that  the  technology  will  have  enough  time  to  mature.  In  other  instances,  a 
higher  than  expected  TRL  may  be  required,  such  as  if  the  technology  in 
question  is  the  linchpin  for  the  entire  product.  Nonetheless,  we  found  that 
TRLs  ably  reconciled  the  different  maturity  levels  and  product  experiences 
of  the  23  technologies  reviewed. 


Technologies  With 
High  Readiness  Levels 
at  Launch  Were  Better 
Able  to  Meet  Product 
Objectives 


The  23  technologies  reviewed  spanned  a  wide  range  of  readiness  levels  at 
the  time  they  were  included  in  product  development  programs.  The  least 
mature  reached  TRL  2  at  the  time  it  was  included  in  a  product 
development,  while  the  most  mature  had  reached  TRL  9  at  the  point  of 
inclusion.  We  observed  a  general  relationship  between  TRLs  and  the 
technologies’  inclusion  on  the  intended  product  developments.  Those 
products  whose  technologies  reached  high  TRLs  at  the  time  they  were 
included  were  better  able  to  meet  cost,  schedule,  and  performance 
requirements.  In  fact,  commercial  firms  informed  us  that  maturing  the 
technology  separately  from  and  ahead  of  the  product  was  a  main  reason 
they  were  able  to  reduce  cycle  times  on  their  products.  An  official  from  one 
of  the  firms  termed  the  approach  as  “moving  discovery  to  the  left.” 


Those  technologies  with  low  TRLs  at  inclusion  encountered  maturation 
difficulties  and  contributed  to  problems  the  products  experienced.  Other 
problems,  such  as  funding  and  schedule  changes  unrelated  to  the 
technologies,  also  contributed  to  problems  in  the  product  developments. 
Figure  2.2  shows  the  TRLs  when  each  of  the  23  technologies  was  included 
in  a  product  design,  whether  at  product  development  launch  (for 
commercial  technologies)  or  at  program  launch  (for  DOD  technologies). 
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Figure  2.2:  Readiness  Levels  of  Technologies  at  the  Time  They  Were  Included  in  Product  Designs 


Non-penetrating  periscope 
Adaptive  cruise  control 
Night  vision 

Voice  activated  control 
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Moving  map  and  navigation 
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The  cost  and  schedule  experiences  of  some  of  the  products  or  programs 
that  inherited  the  technologies  are  shown  in  table  2.1. 


Table  2.1 :  Cost  and  Schedule  Experiences  on  Product  Developments 

Product  development 

TRL  at 

Product  development  and 

program 

associated  technologies 

launch 

Cost  growth 

Schedule  slippage 

Comanche  helicopter 

101  percent3 

120  percent3 

Engine 

5 

Rotor 

5 

Forward  looking  infrared 

3 

Helmet  mounted  display 

3 

Integrated  avionics 

3 

BAT 

88  percent 

62  percent 

Acoustic  sensor 

2 

Infrared  seeker 

3 

Warhead 

3 

Inertial  measurement  unit 

3 

Data  processors 

3 

Hughes  HS-702  satellite 

None 

None 

Solar  cell  array 

6 

Ford  Jaguar 

None 

None 

Adaptive  cruise  control 

8 

Voice  activated  controls 

8 

aThe  Comanche,  in  particular,  has  experienced  a  great  deal  of  cost  growth  and  schedule  slippage  for 
many  reasons,  of  which  technology  immaturity  is  only  one.  Other  factors,  such  as  changing  the  scope, 
funding,  and  pace  of  the  program  for  affordability  reasons,  have  also  contributed. 


Data  for  three  weapon  system  development  programs,  the  Virginia  class 
attack  submarine,  AAAV,  and  the  ABL,  were  not  included  in  the  table 
because  they  had  not  been  in  the  product  development  phase  long  enough 
to  report  actual  consequences.  To  date,  AAAV  and  the  submarine  have 
stayed  within  15  percent  of  their  cost  and  schedule  estimates  for 
development.  The  ABL,  for  which  key  technologies  were  much  less  mature 
at  program  launch,  still  faces  challenges  with  these  technologies.  Ford’s 
night  vision  technology  was  excluded  because  the  firm  decided  not  to 
include  the  technology  on  a  product.  Details  on  the  Comanche,  BAT,  the 
Virginia  class  attack  submarine,  and  Ford  technology  and  product 
experiences  follow. 
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Technology  and  Product 
Experiences  on  Ford  and 
Virginia  Class  Attack 
Submarine 


The  key  technologies  for  the  Ford  Jaguar  and  the  Virginia  class  attack 
submarine  followed  the  pattern  of  increasing  TRLs  until  they  demonstrated 
a  low  risk  for  transition  to  the  product.  Two  examples  are  Ford’s  voice 
activated  controls  development  and  DARPA’s  nonpenetrating  periscope 
development  for  the  submarine.  In  both  cases,  the  technologies  were 
validated,  operational  prototypes  demonstrated,  and  the  technologies  had 
demonstrated  the  form,  fit,  and  function  of  the  final  article  by  the  beginning 
of  product  development. 

Ford’s  voice  activated  controls  technology,  which  allows  a  driver  to  control 
certain  functions  such  as  windows  and  the  radio  through  verbal 
commands,  was  under  development  in  the  technology  base  for  over 
10  years,  being  pushed  by  the  firm’s  technology  leaders.  It  was  not  until 
1993  that  Ford  found  that  (1)  other  complementary  technologies,  such  as 
processor  speeds  and  low  cost  memory,  had  become  available  and 
(2)  customers  wanted  more  features  and  functions  but  less  distractions 
from  driving.  Given  this  market  information,  Ford  decided  to  pursue  voice 
technology  as  a  strategic  technology  in  terms  of  product  differentiation, 
recognizing  the  importance  of  being  first  to  market  with  this  enabling 
technology.  Figure  2.3  shows  the  time  line  for  developing  this  technology. 


Figure  2.3:  Time  Line  for  Ford’s  Development  of  Voice  Activated  Controls  Technology 


Between  1993  and  1994,  based  on  discussions  with  customers,  Ford 
developed  cost  and  performance  requirements  for  the  technology.  Ford  has 
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never  relaxed  them.  By  September  1995,  when  Ford  allowed  the  technology 
into  the  development  program  for  a  new  Jaguar  design,  voice  activated 
controls  had  been  demonstrated  as  an  integrated  system  in  the  appropriate 
form  and  fit  for  the  Jaguar.  Ford  officials  stated  that  the  product  has  met  all 
cost  and  cycle  time  targets  established  at  the  outset  of  its  development. 
Figure  2.4  shows  the  Jaguar. 


Figure  2.4:  Jaguar 


Ford  demonstrated  voice  activated  control  technology  in  the  appropriate  form  and  fit  before 
incorporating  it  into  the  Jaguar. 

Source:  Ford  Motor  Company. 

DARPA  began  developing  the  nonpenetrating  periscope  technology  as  part 
of  its  submarine  technology  development  efforts  after  recognizing,  in  1988, 
along  with  the  Navy,  that  the  nonpenetrating  technology  would  enhance 
operator  visibility,  provide  greater  submarine  design  flexibility,  and  be 
stealthier  than  conventional  masts  and  periscopes.  At  the  time,  the  Virginia 
class  attack  submarine  program  had  not  been  initiated.  Once  the  decision 
was  made  to  include  the  nonpenetrating  periscope,  it  became  a  key  feature 
of  the  submarine  and  was  a  major  design  driver  for  the  submarine’s  overall 
configuration. 

Nonpenetrating  refers  to  the  fact  that  the  periscope  is  essentially  a  group  of 
sensors  that  are  linked  to  the  submarine  via  fiber  optic  and  other  cables. 
This  technology  uses  infrared  imaging  and  advanced  sensors  to  replace 
conventional  periscopes  and  frees  up  physical  space  compared  with  a 
conventional  periscope.  A  conventional  periscope  relies  on  a  series  of 
telescoping  shafts  and  reflecting  surfaces  to  see  above  the  water’s  surface. 
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When  the  periscope  is  retracted,  the  shafts  take  up  a  column  of  space  from 
the  top  of  the  submarine  to  the  bottom,  through  all  decks.  Its  location 
virtually  dictates  the  design  and  placement  of  the  control  and  other  rooms. 
If  the  nonpenetrating  periscope  technology  did  not  become  available,  then 
the  submarine  would  have  to  be  drastically  redesigned  to  accommodate  the 
space  required  by  a  conventional  periscope. 

The  new  nonpenetrating  periscope  and  photonics  mast  technology 
underwent  land  testing  in  1991— a  TRL  5.  The  Navy  actually  tested  the  new 
technology  at  sea  on  the  U.S.S.  Memphis  in  1992  and  1993.  According  to 
program  officials,  these  sea  trials  demonstrated  the  highest  level  of 
technology  readiness:  proving  the  actual  system  through  successful 
mission  operations.  This  readiness  equated  to  a  TRL  9.  Yet,  this  technology 
was  not  included  in  the  Virginia  class  attack  submarine  requirements  until 
1995.  Figure  2.5  shows  an  artist’s  concept  of  the  Virginia  class  attack 
submarine. 


The  Navy  demonstrated  a  key  technology  at  the  highest  readiness  level  before  including  it  as  a 
requirement  for  the  Virginia  class  attack  submarine. 

Source:  DOD. 

The  high  readiness  level  of  the  nonpenetrating  periscope  afforded  the  Navy 
the  opportunity  to  develop  an  improved  version  of  the  periscope  to  a 
TRL  9.  This  was  a  relatively  low-risk  endeavor  as  the  baseline  periscope 
was  sufficient  to  meet  the  submarine’s  requirements.  Program  officials 
believe  that  having  knowledge  about  key  technologies,  such  as  the 
nonpenetrating  periscope,  for  the  Virginia  class  attack  submarine  at 
program  launch  made  a  short  program  definition  and  risk  reduction  phase 
possible.  This  phase  for  the  Virginia  class  attack  submarine  was  about 
75  percent  shorter  than  those  of  previous  acquisition  programs.  Based  on 
its  demonstrated  maturity,  we  anticipate  that  the  nonpenetrating  periscope 
to  be  less  likely  to  impact  the  cost  and  schedule  of  the  submarine’s 
development  program.  There  are,  however,  several  other  technologies  that 
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are  critical  to  the  submarine  program.  We  did  not  examine  these 
technologies  and  cannot  predict  their  likely  outcomes. 


BAT  and  Comanche  Cases  Key  technologies  for  the  BAT  and  Comanche  programs  had  much  lower 

readiness  levels  at  the  time  the  product  developments  were  launched. 
Consequently,  they  did  not  reduce  the  gap  between  their  demonstrated 
maturity  and  the  maturity  needed  to  meet  product  requirements  until  after 
program  launch.  For  some  technologies,  the  gap  was  not  closed  until  well 
into  the  product  development  program.  For  others,  the  gap  has  still  not 
been  closed. 

Five  key  technologies  included  in  the  Army’s  BAT  program  had  low  TRLs 
when  they  were  included  on  the  program.  The  level  of  readiness  for  most  of 
these  technologies  at  program  launch  was  characterized  by  the  program 
office  as  experimental  in  nature  but  with  major  uncertainty  remaining — a 
TRL  3.  The  acoustic  targeting  technology  was  the  most  important  enabling 
technology  needed  to  meet  the  weapon’s  performance  requirements.  This 
technology  provides  BAT  the  capability  to  locate  targets  from  great 
distances  based  on  the  sounds  generated  by  the  target,  such  as  moving 
tanks  and  vehicles.  At  the  time  the  program  was  launched,  the  Army  knew 
little  about  the  feasibility  of  using  this  technology  on  this  program.  In  fact, 
the  technology  was  still  being  defined  in  paper  studies — a  TRL  2.  The 
Army  did  not  prototype  this  technology  until  after  the  program  had  entered 
the  engineering  and  manufacturing  development  phase,  more  than  6  years 
after  program  launch.  As  of  December  1998,  the  BAT  had  experienced 
significant  development  cost  and  schedule  increases,  which  program 
officials  attribute  at  least,  in  part,  to  unknowns  about  the  new  technologies. 
Figure  2.6  shows  the  BAT. 
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Figure  2.6:  Brilliant  Anti-Armor  Submunition 


At  the  time  the  program  was  launched,  the  Army  knew  relatively  little  about  the  performance  of  several 
key  technologies  for  the  BAT 

Source:  DOD. 

Two  technologies  key  to  meeting  the  Comanche  helicopter’s  requirements- 
integrated  avionics  and  forward-looking  infrared  (FLIR)  technologies — 
were  included  on  the  program  when  they  were  still  conceptual  in  nature. 
The  integrated  avionics  technology  replaces  individual  radios,  navigation, 
and  other  communication  equipment  with  a  modular  system  that  shares  a 
common  processor.  The  FLIR  is  a  second-generation  version  that  uses 
infrared  sensors  to  improve  the  pilot’s  ability  to  see  at  night  and  in  bad 
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weather.  Program  officials  stated  that  both  had  TRLs  of  3  when  the 
helicopter  program  was  started.  Despite  the  low  readiness  levels  of  the 
technologies,  the  Army  included  the  technologies  on  the  program  to  meet 
weight,  cost,  and  performance  requirements. 

The  development  of  these  technologies  has  taken  longer  than  the  Army 
expected  it  would.  The  contractor  for  the  integrated  avionics  has  had 
difficulties  in  getting  the  multiple  avionics  modules  to  work  simultaneously 
within  required  size  and  weight  parameters,  and  the  FLIR  technology  has 
undergone  several  design  and  performance  requirement  changes.  As  of 
September  1998 — approximately  10  years  after  program  launch — neither 
the  integrated  avionics  nor  the  FLIR  technology  had  advanced  past  a  TRL  5. 
Problems  with  the  maturation  of  these  technologies  have  contributed  to 
the  program’s  cost  and  schedule  increases.  In  contrast,  the  advanced  rotor 
and  engine  technologies,  which  were  the  most  mature  of  the  Comanche 
technologies  we  reviewed,  have  experienced  fewer  problems  in  maturation 
and  have  not  contributed  significantly  to  the  program’s  cost  and  schedule 
increases.  Figure  2.7  shows  the  Comanche. 


Figure  2.7:  Comanche  Helicopter 


4 


The  Army  included  two  key  technologies  in  the  Comanche  when  they  were  still  considered  conceptual 
to  meet  weight,  cost,  and  performance  requirements 

Source:  DOD. 
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Closing  the  gap  between  technology  maturity  and  product  requirements 
before  a  product  is  launched — and  baselines  are  set — distinguished  the 
more  successful  cases.  Notably,  closing  the  gap  before  product  launch  was 
a  managed  result;  it  put  product  managers  in  a  better  position  to  succeed. 
Two  conditions  were  critical  to  achieving  this  kind  of  result.  First  was  an 
environment  that  put  the  primary  responsibility  for  maturing  technology  in 
the  hands  of  S&T  managers  and  provided  them  considerable  flexibility  to 
make  decisions.  Second  was  having  the  quality  information  and  standards 
needed  to  make  good  technology  handoff  decisions,  coupled  with  giving 
the  product  manager  the  authority  to  refuse  new  technology  that  did  not 
meet  product  requirements.  When  these  conditions  were  not  present,  the 
handoff  to  the  product  manager  was  compromised,  with  negative 
consequences  for  both  technology  and  product. 

In  each  of  the  successful  cases,  S&T  organizations  played  major  roles  in 
bridging  the  gap  between  technology  maturity  and  product  requirements. 
Flexibility  provided  by  requirements  communities  and  resource  providers 
enabled  S&T  and  product  managers  to  delay  the  inclusion  of  technology  if 
it  was  not  ready  or  to  reduce  product  requirements  to  match  what  mature 
technology  could  deliver.  This  environment  was  better  suited  to  the 
unexpected  results  and  delays  that  accompany  technology  development. 
Moreover,  technology  maturation  was  managed  within  a  disciplined 
process  that  provided  good  information  to  be  judged  against  clear  and  high 
standards,  like  TRLs.  Armed  with  the  tools  and  the  authority  to  make 
technology  inclusion  decisions,  both  S&T  and  product  managers 
functioned  as  gatekeepers  to  safeguard  the  product  development. 

In  the  more  problematic  cases,  S&T  organizations  disengaged  much  earlier, 
and  product  managers  had  little  choice  but  to  accept  immature 
technologies.  Accordingly,  less  information  about  the  technologies  was 
available  at  the  point  of  inclusion.  Often,  the  tools  used  to  assess  the 
technologies’  status  failed  to  identify  high  risks.  In  retrospect,  TRLs 
indicated  that  risks  were  in  fact  high  and  perhaps  unacceptable  from  a 
product  standpoint.  Also,  pressures  to  meet  cost  and  schedule  estimates  in 
product  development  provided  a  less  forgiving  environment  for 
technologies  in  the  discovery  process. 
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Providing  the  Right 
Environment  Is  Critical 
to  the  Successful 
Maturation  of 
Technology 


While  most  new  technologies — commercial  and  military — are  initially 
managed  by  the  S&T  community,  the  more  successful  cases  we  reviewed 
continued  to  be  managed  by  S&T  organizations  until  they  reached  at  least 
TRL  6  and  more  often  TRL  8  or  higher.  These  technologies  were  provided 
the  environmental  advantages  an  S&T  project  has  over  a  product 
development.  This  environment  availed  S&T  managers  and  product 
managers  of  the  less  risky  options  of  waiting  or  trading  to  get  the  match 
between  technology  maturity  and  product  requirements — rather  than 
forcing  the  product  launch  and  gambling  on  the  completion  of  technology 
maturity.  In  contrast,  the  more  problematic  technologies  did  not  have  as 
benign  an  environment.  Often,  the  technologies  were  handed  off  early  by 
S&T  organizations  because  inflexible  performance  requirements  for  the 
product  demanded  their  inclusion.  Product  development  managers 
launched  the  product  development  and  hoped  that  the  technology 
development  would  succeed. 


Once  in  a  product  development  environment,  external  pressures  to  keep 
the  program  moving  become  dominant,  such  as  preserving  cost  and 
schedule  estimates  to  secure  budget  approval.  For  example,  DOD  policies 
require  that  a  program  be  funded  in  the  current  year  and  that  funds  be 
made  available  over  the  next  6  years  in  the  DOD  planning  cycle.  If,  dining 
the  program  definition  phase,  a  program  manager  were  to  decide  that  an 
additional  year  was  needed  to  overcome  unexpected  technology  problems 
to  reach  the  desired  level  of  maturity,  the  delay  could  push  the  start  of 
engineering  and  manufacturing  development  back.  This  delay  could 
jeopardize  the  funding  for  that  phase,  thus  risking  the  funding  support  for 
the  entire  program.  Consequently,  the  program  manager  may  be  more  likely 
to  accept  the  risk  of  not  getting  the  technology  to  the  desired  level  of 
maturity  and  starting  the  engineering  and  manufacturing  development 
phase  as  planned,  rather  than  risk  the  rest  of  the  program.  These  conditions 
compete  with  and  detract  from  the  needs  of  technology  development.  One 
acquisition  official  stated  that  these  conditions  cause  the  weapon  system 
program  “to  pull  double  duty,”  inventing  new  technology  while  integrating 
it  into  a  product.  In  general,  he  believed  there  is  an  equal  amount  of 
difficulty  in  both  tasks. 


Technologies  Matured  by 
S&T  Organizations  Made 
Smooth  Transitions  into 
Product  Developments 


In  the  most  successful  cases  that  we  reviewed,  S&T  organizations  bridged 
the  gap  between  immature  technology  and  the  maturity  needed  for  either 
program  start  in  DOD  (TRL  6)  or  product  development  (TRL  7  or  higher). 
These  cases  and  the  responsible  organizations  are  shown  in  table  3.1. 
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Table  3.1 :  TRLs  of  Technologies  Managed  by  S&T  Organizations 

Technology 

TRL  at 
handoff 

Responsible 

S&T  organization 

Receiving  product 
development  program 

Nonpenetrating 

periscope 

9 

DARPA 

Virginia  class  attack 
submarine  program 

Adaptive  cruise  control 

8 

Ford  Advanced  Vehicle 
Technology  Office 

Jaguar  vehicle  team 

Voice  activated  controls 

8 

Ford  Advanced  Vehicle 
Technology  Office 

Jaguar  vehicle  team 

Solar  cell  array 

6 

Hughes  Laboratories 

HS-702  satellite 
program 

Weapon  ejection 
system 

6 

Office  of  Naval 

Research 

Virginia  class  attack 
submarine  program 

Diesel  powered  engine 

6 

Office  of  Naval 

Research 

AAAV  program 

High-speed  planing 
craft 

6 

Office  of  Naval 

Research 

AAAV  program 

High-power  water  jet 

6 

Office  of  Naval 

Research 

AAAV  program 

Despite  the  different  circumstances  between  the  commercial  and  DOD 
sectors  and  among  the  DOD  cases  themselves,  the  results  were  similar: 
having  S&T  organizations  bridge  the  maturity  gap  reduced 
technology-related  problems  in  the  products.  For  the  leading  commercial 
firms  we  visited,  it  is  standard  practice  to  have  S&T  organizations 
responsible  for  the  bridge.  In  the  DOD  cases  shown  in  table  3.1,  the  S&T 
organizations  played  atypical  roles  in  managing  the  bridge  between 
technology  and  product  by  delivering  the  technology  to  a  TRL  6  or  higher. 
Different  pressures  and  incentives  that  are  brought  to  bear  on  the 
commercial  and  DOD  product  developments  explain  why  DOD  product 
managers  become  responsible  for  more  technology  development  than  their 
commercial  counterparts.  These  influences  are  discussed  in  chapter  4. 

Having  an  S&T  organization  manage  a  technology  to  maturation  means 
more  than  just  having  a  different  group  of  people  involved  than  a  product 
development.  S&T  projects  operate  in  a  different  environment  than 
product  developments.  The  process  of  developing  technology  culminates 
in  discovery  and  must,  by  its  nature,  allow  for  unexpected  results.  S&T 
provides  a  more  forgiving  environment  in  which  events — such  as  test 
“failures,”  new  discoveries,  and  delays  in  the  attainment  of  knowledge — are 
considered  normal.  It  is  also  a  less  costly  environment,  making  external 
pressures  to  develop  knowledge  on  a  schedule  less  keenly  felt.  On  the 
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other  hand,  the  process  of  developing  a  product  culminates  in  delivery,  and 
thus  gives  great  weight  to  design  and  production.  The  same  events  and 
unexpected  results  that  are  considered  normal  for  technology  development 
represent  problems  in  the  product  environment;  they  can  jeopardize 
achievement  of  cost  and  schedule  objectives  and  draw  criticism  to  the 
product.  The  ups  and  downs  and  the  resource  changes  associated  with  the 
technology  discovery  process  do  not  mesh  well  with  a  program’s  need  to 
meet  cost,  schedule,  and  performance  goals.  This  situation  has  been 
described  as  attempting  to  “schedule  inventions.” 


Successful  Cases  Afforded  In  the  early  1980s,  Hughes  Space  and  Communications  began  developing 
Flexibility  to  dual  junction  solar  cell  technology  that  had  the  potential  of  greatly 

Decisionmakers  increasing  the  electrical  power  on  satellites.  By  1985,  a  Hughes  laboratory 

had  demonstrated  the  technology  by  ground  testing  prototypes,  a  TRL  6, 
which  is  considered  an  acceptable  level  of  demonstration  for  space-based 
technology.  Nonetheless,  Hughes  was  not  satisfied  that  the  supporting 
infrastructure  (materials,  reactors,  and  test  equipment)  was  mature  enough 
to  sustain  development  and  production  of  the  new  technology  on  a 
satellite.  The  infrastructure  was  seen  as  critical  to  meeting  the  cost  and 
schedule  requirements  of  a  product.  As  a  result,  Hughes  did  not  hand  off 
the  technology  to  a  product.  Instead,  the  firm  kept  it  in  a  research 
environment,  away  from  cost  and  schedule  pressures. 

In  the  early  1990s,  Hughes  established  requirements  for  a  new  satellite — 
the  HS-702 — that  would  use  the  solar  cell  technology  to  leapfrog  the 
competition.  After  a  laboratory  demonstration  in  1993,  Hughes  successfully 
used  the  new  technology  on  a  high-powered  version  of  its  existing  HS-601 
satellite  before  it  began  product  development  on  the  HS-702  satellite.  By 
1994,  it  had  determined  that  the  business  base  was  available  to  sustain 
development  and  production  of  the  HS-702  satellite.  In  all,  the  firm  waited 
10  years  for  the  demonstrated  technology  to  meet  the  requirements.  This 
experience  closely  resembled  that  of  Ford’s  voice  activated  control 
technology  because,  in  both  cases,  the  new  technology  took  10  years  to 
mature  enough  for  product  readiness.  Thus,  the  firms’  approach  was  not  to 
accelerate  technology  development  but  to  shorten  product  development  by 
maturing  the  technology  first.  Figure  3.1  shows  the  solar  cell  arrays 
installed  on  the  HS-702. 
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Figure  3.1 :  Hughes  Solar  Cell  Arrays 


Hughes  successfully  proved  solar  cell  array  technology  on  a  predecessor  satellite  before  beginning 
product  development  of  the  HS-702 

Source:  Hughes  Space  and  Communications. 
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The  Navy  made  trade-offs  in  choosing  a  technology  for  the  weapon  ejection 
system,  which  is  used  to  deploy  weapons  like  torpedoes,  of  the  Virginia 
class  attack  submarine.  Because  of  quietness,  weight,  and  cost 
requirements,  the  Navy  preferred  a  new  elastomeric  (rubber-based) 
technology.  However,  this  technology  failed  endurance  testing,  and  product 
managers  determined  that  the  technology  was  too  risky  to  be  included  in 
the  first  product.  Product  managers  could  have  declined  this  technology 
and  its  attendant  risk  without  delaying  the  submarine’s  schedule  because 
the  Navy  accepted  marginal  increases  to  the  cost  and  weight  requirements 
for  the  system  so  that  the  proven  Seawolf  ejection  technology  could  be 
used  as  a  substitute.  Using  proven  technology  on  the  first  submarine  has 
allowed  the  Navy  S&T  community  to  continue  developing  the  elastomeric 
technology,  which  is  to  be  incorporated  into  the  new  system  on  the  fourth 
production  submarine.  As  discussed  earlier,  decisionmakers  also  had  the 
flexibility  to  wait  for  the  nonpenetrating  periscope  technology  to  reach 
TRL  9  before  including  it  in  the  submarine’s  requirements. 


Problematic  Cases  Provided 
Little  Flexibility  to 
Managers 


According  to  Army  officials,  the  FUR  and  integrated  avionics  technologies 
required  for  the  Comanche  helicopter  were  critical  for  providing  an 
increased  operational  capability  over  existing  Army  helicopters.  The 
advanced  FUR  technology  was  needed  to  meet  the  user’s  requirements  for 
increased  targeting  range  and  for  improved  piloting  capabilities  in  bad 
weather  and  at  night.  It  represented  a  quantum  leap  from  existing 
capabilities.  Integrated  avionics  technology  was  expected  to  replace 
separate  radios,  navigation  systems,  and  other  communication  equipment 
on  the  helicopter  with  a  modular  system  that  uses  central  processors. 


These  technologies  were  needed  to  meet  weight  and  size  requirements  for 
the  aircraft  as  well  as  improve  communications.  Both  were  critical 
elements  of  a  mission  equipment  package  that  was  supposed  to  reduce  the 
pilot’s  workload  while  improving  capabilities.  Requirements  were 
inflexible.  Thus,  requirements  managers  informed  us  they  were  unwilling 
to  accept  the  product  manager’s  request  to  trade  requirements  that  was 
prompted  by  his  concerns  that  the  technologies  could  not  advance  in  time 
to  meet  the  program’s  schedule.  They  believed  the  product  manager  to  be 
too  risk  averse  and  said  they  would  not  take  no  for  an  answer.  Not  only  did 
the  user  consider  the  technologies  nontradable,  they  became  even  more 
confined  by  weight  and  cost  restrictions  that  were  placed  on  the  program. 
For  example,  a  more  mature  FUR  technology  that  could  possibly  meet 
performance  requirements  but  also  weighed  more  was  rejected. 


Page  39 


GAO/NSIAD-99-162  Best  Practices 


Chapter  3 

Controllable  Conditions  Affect  How  Well  a 
Technology’s  Inclusion  on  a  Product  Can  Be 
Managed 


The  technological  solutions  that  could  meet  the  strict  requirements  were 
limited.  According  to  Army  officials,  the  only  viable  option  was  to  develop 
the  new  technologies,  which  were  in  a  very  immature  state,  to  the  required 
performance  levels  because  no  suitable  back-up  technologies  existed. 
When  the  Comanche  acquisition  program  was  launched,  the  FLIR  and 
integrated  avionics  technologies  had  a  TRL  3,  barely  demonstrated  in  a 
laboratory.  This  level  placed  the  burden  on  the  Comanche  program 
manager  to  complete  their  development  during  the  acquisition  program. 
The  only  ways  for  the  program  manager  was  to  slip  the  schedule  or 
increase  development  costs.  Figure  3.2  shows  an  early  model  of  the 
integrated  avionics  component  for  the  Comanche. 


Figure  3.2:  Integrated  Avionics  for  Comanche  Helicopter 


The  Army  launched  the  Comanche  program  with  immature  technologies,  placing  the  burden  on  the 
program  manager  to  complete  technology  development 

Source:  DOD. 
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Similarly,  the  acoustic  sensor  technology  on  the  BAT  was  critical  to  the 
submunition’s  performance  because  it  provided  breakthrough 
improvements  in  the  capability  for  precision  attack  of  targets  at  ranges  of 
up  to  500  kilometers  and  in  most  weather.  There  was  no  flexibility  for  the 
program  manager  to  ease  requirements  to  substitute  a  more  mature 
technology  because  the  Army  had  no  existing  capability  to  perform  this 
mission.  Thus,  the  technology,  which  had  a  TRL  2  at  program  launch,  was 
the  only  solution  for  locating  and  acquiring  targets.  Its  feasibility  was  based 
on  an  engineering  analysis  in  the  form  of  studies.  Key  challenges  for  the 
acoustic  sensor  were  to  reduce  noise  to  an  acceptable  level,  develop 
microphones  with  sufficient  range,  and  reduce  the  size  of  the  sensor  so  it 
would  fit  into  the  BAT  delivery  system.  The  technology  development  that 
was  necessary  to  have  the  sensor  meet  requirements  had  to  be 
accomplished  during  the  schedule-driven,  delivery-oriented  product 
development  program.  The  development  program  encountered  technical 
problems  that  left  the  program  manager  with  no  choice  but  to  slip  the 
schedule  and  increase  the  cost.  By  the  start  of  the  engineering  and 
manufacturing  development  phase,  program  officials  stated  that  the 
acoustic  sensor  had  a  TRL  5 — still  a  high  risk  using  the  Air  Force  Research 
Laboratory’s  criteria. 


Good  Technology 
Handoff  Decisions 
Depend  on  the  Tools 
and  Authority  Given  to 
Managers 


With  the  right  environment  as  a  precondition,  managers  on  the  successfiil 
cases  benefited  from  disciplined  technology  development  processes  that 
linked  the  technologies  to  products  and  provided  credible  information  on 
the  status  of  technologies.  They  also  had  standards  that  were  both  clear 
and  high  for  assessing  readiness.  Once  a  technology’s  feasibility  and 
usefulness  were  demonstrated,  it  was  linked  to  a  product  through  an  early 
agreement  with  the  product  developer  to  use  it  if  it  could  be  fully 
developed.  Ideally,  as  technologies  approached  the  higher  readiness  levels 
associated  with  the  bridge,  S&T  managers  and  receiving  product  managers 
agreed  to  more  specific  terms  for  accepting  or  rejecting  a  technology. 
These  agreements  were  early  links  to  the  product  that  were  needed  for  the 
technology  to  succeed.  If  a  product  manager  was  not  willing  to  make  such 
an  agreement,  then  the  investment  to  bring  the  technology  to  higher 
readiness  levels  might  not  be  made.  S&T  managers  were  responsible  for 
ensuring  that  information  at  key  junctures  was  sufficient  and  that  the 
technology  was  ready  for  inclusion  on  a  product.  They  saw  their  role  as  to 
screen  and  develop  technologies  to  standards  acceptable  to  product 
managers.  Product  managers  were  responsible  for  ensuring  that  the 
product  could  be  developed  and  brought  to  market  within  cost  and 
performance  targets.  They  saw  as  their  role  to  encourage  the  successful 
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development  of  new  technology  but  to  decline  the  handoff  if  it  did  not  meet 
product  performance,  cost,  and  schedule  requirements. 

When  an  S&T  organization  disengages  from  a  technology  at  a  low  TRL,  the 
S&T  manager  gives  up  much  of  the  ability  to  be  a  gatekeeper.  In  the  event 
that  unyielding  requirements  or  other  pressures  force  product  managers  to 
accept  technologies  before  they  have  matured,  they  are  weakened  in  their 
ability  to  safeguard  the  product  development  from  technology  risks.  For 
the  cases  in  which  technologies  had  problems  transitioning  to  products, 
decisionmakers  were  disadvantaged  by  the  incomplete  information 
available  to  them,  yet  were  not  empowered  to  say  no  to  the  handoff.  Their 
situation  was  further  degraded  by  risk  assessments  that  embodied  lower 
standards  for  accepting  undemonstrated  technology  readiness.  In  the  case 
of  the  BAT,  the  S&T  community  was  bypassed  altogether,  as  the  weapon 
system  and  its  enabling  technologies  were  proposed  by  a  contractor  and 
assigned  directly  to  a  program  manager. 


Successful  Cases  Benefited 
From  Strong  Gatekeepers, 
Disciplined  Processes,  and 
High  Maturity  Standards 


All  new  technologies  at  Ford,  regardless  of  whether  they  are  proposed  by 
inside  or  outside  sources,  take  essentially  the  same  path  and  gates  into 
products.  Initially,  technology  proposals  pass  through  a  process  that 
prioritizes  them  according  to  customer  needs.  The  proposals  are  then 
passed  on  to  the  Advanced  Vehicle  Technology  Office,  an  S&T  organization 
that  determines  the  readiness  of  the  proposed  technology  and  fits  it  into 
Ford’s  path  of  technology  demonstration.  Once  approved,  the  technology 
follows  a  structured  process  that  includes  two  development  phases: 
concept  ready  and  implementation  ready.  This  process  results  in  a  smooth 
transition  from  the  technology  development  environment  into  a  product, 
once  the  technology  is  mature.  Ford’s  adaptive  cruise  control  technology 
went  through  this  process,  as  shown  in  figure  3.3. 
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Figure  3.3:  Process  for  Closing  the  Gap  Between  the  Readiness  of  Adaptive  Cruise  Control  Technology  and  Jaguar 
Requirements 
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According  to  Ford  officials,  technologies  for  adaptive  cruise  control 
existed  as  separate  projects  in  the  technology  base  from  about  1993  to 
1995,  when  the  Jaguar  vehicle  team  identified  a  strong  demand  for  the 
capability.  Ford’s  S&T  community  inventoried  the  ongoing  projects  and 
demonstrated  the  technology  as  a  laboratory  breadboard— a  TRL  5.  By 
August  1996,  the  technologists  had  built  a  prototype  that  could 
demonstrate  the  technology  in  a  relevant  environment — a  TRL  7.  This  work 
comprised  the  concept  ready  phase,  in  which  the  technology  was  taken 
from  concept  to  where  its  feasibility  was  demonstrated  to  potential  users. 
At  the  end  of  this  phase,  S&T  representatives  proved  that  it  could  work, 
and  cost,  schedule,  and  performance  targets  were  established.  Also,  a 
target  product  and  sponsor  were  identified,  linking  the  technology  to  a 
product.  The  sponsor  agreed  that  it  would  accept  this  technology  if  specific 
cost,  quality,  schedule,  and  performance  targets  were  met.  The  medium  for 
this  acceptance  was  the  Deliverables  Agreement  Log  (DEAL),  which  was 
signed  in  September  1996  by  Jaguar’s  chief  engineer  based  on  the  prototype 
demonstration. 

Ford  uses  the  DEAL  as  a  tool  to  maintain  visibility  over  a  new  technology 
as  it  progresses  through  the  development  process  and  to  assess  its 
readiness  and  acceptability  for  inclusion  in  a  vehicle  program  before 
handing  it  to  a  sponsor,  the  vehicle  center,  or  team.  The  DEAL  formalizes 
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the  content  of  the  two  development  phases  and  establishes  agreements 
between  the  technologists  managing  the  project  and  those  with  authority  to 
accept  the  technology  into  a  product.  According  to  Ford  officials,  the  DEAL 
is  important  to  this  process  because  it  is  a  contract  between  the  parties  that 
addresses  the  technology’s  performance,  cost,  quality,  weight,  producibility, 
and  maintainability  targets  that  must  be  met  before  the  end  of  each  phase. 
It  has  been  invaluable  in  getting  parties  to  agree  on  what  is  expected  by  the 
giver  and  receiver  of  a  technology  during  the  process. 

Once  these  targets  are  established,  the  technology  moves  to  the 
implementation  ready  phase.  For  the  Jaguar,  the  Advanced  Vehicle 
Technology  Office  matured  the  technology  to  a  high  level  of  readiness  by 
prototyping  it  in  demonstrator  vehicles — a  TRL  8.  The  technology  passed 
the  implementation  ready  milestone  in  February  1997.  At  that  point,  the 
vehicle  team  accepted  the  technology  for  inclusion  on  a  Jaguar  product 
development. 

Ford  used  this  decision-making  process  to  develop  the  night  vision 
technology,  but  with  a  different  result.  Since  1991,  Ford  has  been  working 
on  this  technology  to  provide  a  wide  field  of  view  and  depth  perception  for 
the  driver  at  night,  similar  to  that  provided  by  a  FLIR.  By  1998,  the 
Advanced  Vehicle  Technology  Office  brought  the  technology  to  a  TRL  8. 
However,  the  vehicle  center  did  not  agree  to  include  the  technology  on  a 
product  because  the  technology  did  not  meet  the  cost  targets  established  in 
the  DEAL. 

Other  companies  we  visited  had  similar  practices  for  supporting 
technology  inclusion  decisions.  For  example,  3M  takes  technology  from  its 
technology  base  when  it  believes  it  has  a  customer  need.  The  gatekeeper 
responsible  for  moving  technology  into  a  concept  phase — analogous  to 
TRL  3  or  4— is  the  S&T  organization  of  a  business  unit.  That  business  unit 
monitors  the  technology’s  progress  until  a  new  product  requirement  is 
identified  and  decides  whether  there  is  interest  from  a  product  center  to 
“pull”  it.  If  an  interest  exists,  it  begins  a  feasibility  phase  that  refines 
requirements  through  quality  functional  deployment  and  builds  working 
prototypes  of  the  new  product — a  stage  that  would  be  analogous  to  TRL  7 
or  8.  This  phase  culminates  with  an  agreement  between  the  technologists 
and  the  product  developers — the  receivers — as  to  the  specific  cost  and 
schedule  targets  that  must  be  met  for  the  technology  to  be  included  into  a 
product.  To  help  facilitate  the  transition,  3M  establishes  a  product 
development  team  that  includes  people  from  research  and  development, 
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marketing,  manufacturing,  and  other  functions  that  transfer  with  the  new 
technology  and  ensure  it  is  integrated  into  the  new  product. 

3M  also  has  high  standards  for  measuring  the  readiness  of  a  technology 
before  the  product  developer  accepts  it.  For  example,  3M  officials  told  us 
that  they  are  developing  a  fuel  cell  technology  for  which  they  have  built  15 
prototypes  for  testing  purposes— a  TRL  7  or  higher.  However,  because  the 
technology  has  not  yet  met  all  of  the  cost,  schedule,  and  performance 
targets  for  product  development,  they  have  not  allowed  it  to  be  included  on 
a  new  product,  despite  demand  from  the  marketplace. 

Among  the  DOD  cases,  the  process  followed  and  the  roles  played  on  the 
AAAV  program  had  several  features  that  enabled  good  technology  inclusion 
decisions.  For  almost  3  decades,  the  Marine  Corps  has  stated  a  need  for  an 
amphibious  vehicle  with  far  greater  capabilities  than  the  current  vehicle. 
Specifically,  the  requirement  to  achieve  a  speed  of  20  to  25  knots  in  the 
open  ocean  made  advances  in  propulsion  technology  key  enablers  for  the 
AAAV  program.  For  a  vehicle  of  the  planned  size  and  weight  of  the  AAAV, 
this  requirement  meant  achieving  2,700  horsepower  with  a  relatively 
compact  engine  that  must  operate  on  land  and  in  water.  The  Corps  had 
been  exploring  propulsion  technologies  for  such  a  vehicle  in  its  technology 
base  for  many  years.  Despite  this,  the  Office  of  Naval  Research,  an  S&T 
organization,  assessed  the  propulsion  technology  and  advised  that  it  was 
not  mature  enough  to  warrant  inclusion  on  a  program.  Based  on  this 
assessment,  Marine  Corps  and  Navy  decisionmakers  delayed  program 
launch  from  1991  to  1995,  until  the  technology  could  be  brought  to  higher 
readiness  levels.  Figure  3.4  illustrates  the  process  used  to  transition  this 
technology. 


Page  45 


GAO/NSIAD-99-162  Best  Practices 


Chapter  3 

Controllable  Conditions  Affect  How  Well  a 
Technology’s  Inclusion  on  a  Product  Can  Be 
Managed 


Figure  3.4:  Process  for  Closing  the  Gap  Between  the  Readiness  of  Propulsion  Technologies  and  A  AAV  Requirements 
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The  S&T  community  and  the  product  managers  agreed  on  what  had  to  be 
done  before  the  program  could  be  launched.  The  S&T  community  then 
took  the  lead  in  maturing  the  engine  to  a  TRL  6 — a  level  the  Air  Force 
Research  Laboratory  considers  acceptable  for  starting  the  program 
definition  and  risk  reduction  phase.  Thus,  the  assessment  by  the  Office  of 
Naval  Research  provided  both  the  information  and  the  criteria  that  enabled 
decisionmakers  to  say  no  to  launching  the  program  given  the  low  readiness 
of  the  propulsion  technology.  This  was  coupled  with  the  flexibility  to  wait 
for  the  technology  to  mature  and  the  decision  to  give  an  S&T  organization 
responsibility  for  managing  the  bridge  to  product  readiness.  Figure  3.5 
shows  the  AAA V. 
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The  Marine  Corps  and  Navy  delayed  program  launch  by  4  years  to  develop  key  technologies  to  a 
higher  readiness  level 

Source:  DOD. 

Even  with  an  urgent  need  for  the  AAAV,  the  Marine  Corps  remained 
disciplined  in  its  development  approach,  allowing  the  technology  to  mature 
to  the  level  of  the  requirement.  Two  years  before  program  launch,  a  Navy 
S&T  organization  demonstrated  the  technology  in  a  full-scale  prototype 
engine.  By  program  launch  in  1995,  the  required  2,700  horsepower  was 
demonstrated  by  a  near  prototype  engine — a  TRL  6.  The  remaining  risk  was 
limited  to  marginal  weight  and  size  reductions,  although  the  demonstrator 
engine  could  be  used  as  a  backup  if  the  size  and  weight  reductions  could 
not  be  obtained.  In  early  1999,  the  AAAV  program  office  demonstrated  a 
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prototype  engine  at  2,700  horsepower  that  met  size  and  weight 
requirements — a  TRL  7. 


Technology  Handoffs  Were 
Compromised  When 
Managers  Had  Limited 
Information  and  Authority 


In  the  BAT  program,  neither  the  S&T  community  nor  the  product  manager 
had  the  opportunity  to  act  as  a  gatekeeper  between  product  requirements 
and  the  maturity  of  enabling  technologies.  All  of  the  technologies  for  the 
BAT  came  to  the  program  after  the  contractor,  in  1985,  had  proposed  a 
weapon  concept  for  carrying  out  unmanned,  deep  strike  missions  to  attack 
enemy  armored  vehicles.  Army  leadership  accepted  the  concept  and 
drafted  requirements  for  the  BAT,  and  the  acquisition  program  was 
launched  after  the  proposal  was  accepted.  Thus,  the  technology  for  the 
weapon  came  directly  from  the  contractor’s  technology  base  into  the 
acquisition  program,  with  little  or  no  review  by  the  Army’s  S&T 
organization.  The  process,  information,  and  standards  that  were  critical  to 
successful  technology  inclusion  decisions  in  other  cases  were  not 
employed  on  the  BAT.  The  process  followed  is  shown  in  figure  3.6. 


Figure  3.6:  Assimilation  of  New  Technology  Into  the  BAT  Program 
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The  program  office  accepted  the  acoustic  sensor,  infrared  seeker,  and 
navigation  technologies  included  on  the  BAT  program.  In  retrospect,  the 
levels  of  demonstration  at  the  time  posed  high  risks  to  the  product 
development  because  the  acoustic  sensor  technology  had  a  TRL  of  2  and 
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the  infrared  seeker  and  navigation  technologies  had  TRLs  of  3.  Program 
officials  stated  that  a  significant  amount  of  technology  development  was 
required  during  product  development  due  to  the  lack  of  visibility  over 
technology  readiness  before  program  launch.  As  a  result,  the  development 
program’s  cost  and  schedule  significantly  increased  over  original  estimates. 

An  interesting  sidelight  to  the  BAT  experience  concerns  the  inertial 
measurement  unit,  a  navigation  component  of  the  submunition.  When  the 
contractor  first  proposed  the  BAT  concept,  the  design  included  a  mature 
inertial  measurement  unit  in  production  on  other  systems.  However,  after 
the  program  was  launched,  the  contractor  substituted  a  new  quartz  rate 
technology.  At  the  request  of  the  BAT  program  manager,  the  Army’s  Missile 
Research  and  Development  Engineering  Center,  an  S&T  organization, 
assessed  the  maturity  of  the  quartz  rate  technology.  The  Center  concluded 
that  the  new  technology  had  not  demonstrated  a  high  enough  level  of 
readiness  and  recommended  that  a  more  proven  existing  technology  be 
used  in  the  program.  Eventually,  the  new  technology  was  dropped,  and  an 
existing  technology  that  was  at  a  higher  readiness  level  was  used. 

We  observed  additional  cases  in  which  decisionmakers  relied  on 
comparatively  low  standards  for  including  technologies.  The  Army 
assessed  the  FLIR,  integrated  avionics,  and  helmet  mounted  display 
technologies  as  having  moderate  risk  when  they  were  included  in  the 
Comanche  program.  Army  officials  stated  that  they  required  only  the 
existence  of  an  ongoing  S&T  technology  project  as  acceptable,  as  long  as 
the  technology  was  projected  to  be  ready  by  the  engineering  and 
manufacturing  development  phase.  According  to  program  officials, 
demonstrated  maturity  was  considered  but  not  required;  proof  that  the 
projects  were  progressing  as  scheduled  was  enough.  These  technologies, 
however,  had  TRLs  of  3  at  the  time  of  launch— a  high  risk  for  the  program 
definition  and  risk  reduction  phase.  This  risk  assessment  is  more 
consistent  with  the  actual  experience  of  the  technologies’  maturation  in  the 
program. 

The  standards  used  for  accepting  the  laser  technology  into  the  ABL 
program  also  appeared  low  when  compared  with  the  standards  used  on  the 
more  successful  cases.  While  the  Air  Force  had  established  demonstration 
standards  for  the  laser  to  meet  prior  to  program  launch,  these  standards 
were  met  if  scale  models  of  the  laser  technology  in  a  laboratory 
demonstrated  they  had  the  potential  to  produce  the  energy  needed  for  an 
operational  system.  This  level  of  technical  demonstration  equated  to  a  TRL 
of  4,  representing  a  high  risk  for  inclusion  into  an  acquisition  program. 
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Although  product  developments — commercial  or  defense — fare  better 
when  key  technologies  are  matured  before  they  are  included  in  the  product 
design,  the  more  traditional  approach  within  DOD  is  to  mature  technology 
during  a  product’s  development.  Rational  explanations  are  behind  this 
tradition.  S&T  organizations,  operating  within  fixed  budget  levels,  are  not 
necessarily  accustomed  or  equipped  to  manage  the  bridge  between 
technology  feasibility  and  product  readiness.  Programs  are  more  able  to 
command  the  large  budgets  necessary  for  reaching  higher  levels  of 
technology  readiness  than  S&T  projects.  Also,  pressures  are  exerted  on 
new  programs  to  offer  unique  performance  and  acceptable  cost  and 
schedule  projections,  which  encourage  premature  acceptance  of  unproven 
technologies. 

The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  not  only 
supports  shorter  cycle  times  and  a  more  aggressive  pursuit  of  technology 
outside  of  programs,  but  also  use  of  commercial  best  practices  to  get  these 
results.  DOD  has  several  initiatives  underway  that  could  make  conditions 
more  favorable  for  S&T  organizations  to  mature  a  technology  further 
before  it  is  included  in  a  product  development.  One  Army  project  calls  for 
an  S&T  organization  to  manage  all  technology  maturation  and  integration 
tasks  for  a  new  combat  vehicle  up  to  the  engineering  and  manufacturing 
development  phase.  Other  initiatives  may  make  the  S&T  community  a  more 
integral  participant  in  matching  user  requirements  with  technology  and 
tying  S&T  projects  more  closely  to  product  development  paths.  Whether 
these  efforts  are  effective  and  can  be  applied  on  a  broader  scale  remains  to 
be  seen. 


Several  Factors  Make 
It  Difficult  to  Mature 
Technologies  Before 
They  Are  Included  on 
Weapon  Systems 


Budgetary,  organizational,  and  other  factors  within  DOD  make  it  difficult  to 
bring  technologies  to  high  readiness  levels  before  being  included  in 
weapon  systems.  These  factors  encourage  S&T  organizations  to  disengage 
from  technology  development  too  soon  and  weapon  system  program 
managers  to  accept  immature  technology.  Factors  other  than  these 
encourage  leading  commercial  firms  to  keep  technology  development  out 
of  the  product  developers’  hands  and  in  those  of  S&T  organizations.  The 
differences  in  these  factors  and  in  the  management  of  technology 
development  stem  from  differences  in  what  helps  commercial  and  DOD 
programs  to  succeed.  They  do  not  stem  from  capabilities  commercial  firms 
possess  that  DOD  does  not. 
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Budget  realities  within  DOD — the  fact  that  weapon  system  programs 
attract  higher  levels  of  funding  than  S&T  projects — make  these  programs  a 
more  advantageous  setting  for  funding  technology  development  to  the 
higher  readiness  levels.  As  a  practical  matter,  it  is  often  necessary  to  move 
immature  technology  to  a  weapon  system  program  to  get  needed  funds  and 
management  support  for  maturation.  Normally,  DOD  S&T  organizations  do 
not  see  their  role  as  going  beyond  demonstrating  the  feasibility  of  a 
technology  for  generic — versus  product  specific — application  (a  TRL  5). 
However,  as  seen  in  several  of  the  cases  we  reviewed,  even  this  level  often 
is  not  reached  before  a  product  development  organization  takes  over.  The 
S&T  organizations  that  helped  to  bridge  the  gap  from  technology  feasibility 
to  product  readiness  on  the  more  successful  cases  had  gone  beyond  their 
typical  role. 

One  of  the  reasons  that  S&T  organizations  disengage  relatively  early  is  that 
S&T  work  is  traditionally  funded  as  a  percentage  of  the  overall  DOD 
research  and  development  budget.  S&T  organizations  receive  about 
$8  billion  annually,  or  about  20  percent,  of  DOD’s  research  and 
development  budget.  This  money  funds  several  thousand  projects, 
providing  less  than  $1  million  per  project  on  average.  As  a  result,  a  project 
needing  $100  million  or  more  to  mature  technology  to  higher  readiness 
levels  than  normal — not  unreasonable  sums — would  command  a  fairly 
large  share  of  an  S&T  organization’s  budget,  thereby  reducing  funds 
available  for  other  projects.  Under  the  current  scenario,  the  remaining 
80  percent  of  DOD’s  research  and  development  funds,  approximately 
$30  billion,  is  spread  out  over  a  much  smaller  number  of  specific  weapon 
programs.  A  typical  weapon  system  program  can  receive  several  hundred 
million  dollars  annually  and  occasionally  over  $1  billion  to  fund 
development.  A  nuyor  program,  such  as  the  F-22,  can  command  $15  billion 
or  more  in  total  for  product  development,  receiving  sometimes  more  than 
$2  billion  in  a  year. 

Events  on  the  Air  Force’s  ABL  program  illustrate  these  realities.  Originally, 
the  Air  Force  had  planned  the  ABL  as  a  technology  development  project  to 
be  managed  to  high  readiness  levels  by  an  S&T  organization.  The  project 
was  started  in  1992  as  an  advanced  technology  transition  demonstration  to 
design,  fabricate,  and  test  a  single  demonstrator  weapon  system  and  was  to 
take  8  years  to  complete.  The  pacing  technologies,  the  laser  and  the  beam 
control,  were  to  be  matured  to  a  high  level — equivalent  to  TRL  6  or  7 — 
before  being  included  in  a  product  development  program.  Requirements 
had  not  been  fixed.  In  other  words,  the  planned  approach  resembled  what 
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we  have  described  as  the  more  successful  cases  in  our  review.  Figure  4.1 
shows  the  ABL. 


Figure  4.1 :  Airborne  Laser 


The  Air  Force  used  relatively  low  readiness  level  standards  to  include  a  key  technology  into  ABL 
Source:  DOD. 


In  1996,  the  Air  Force  abandoned  this  approach  and  decided  to  launch  ABL 
as  a  weapon  system  development  program,  not  because  technologies  were 
sufficiently  mature  but  because  of  funding  and  sponsorship  concerns.  At 
this  time,  the  two  key  technologies  were  at  TRLs  3  and  4.  According  to  the 
retired  manager  of  the  S&T  project,  a  product  development  program  was 
deemed  necessary  to  make  the  technology  development  effort  appear  real 
to  the  users  and  not  a  scientific  curiosity.  Within  the  Air  Force,  the 
perceived  lack  of  support  by  the  users  placed  the  project  in  a  constant  state 
of  funding  jeopardy.  This  perception  was  important  because  the  S&T 
project  was  costly,  with  a  total  estimated  cost  of  $800  million,  with  some 
annual  funding  requirements  approaching  $200  million.  The  annual  funding 
requirements  would  encompass  a  large  percentage  of  the  Air  Force’s 
S&T  budget  unless  additional  funds  were  made  available  from  weapon 
system  budgets  or  elsewhere.  By  transitioning  to  a  weapon  system  program 
linked  to  user  requirements,  the  ABL  was  more  likely  to  get  these  funding 
levels. 

This  approach  was  successful — the  program  won  user  support  and  the 
desired  funding.  However,  sacrifices  were  made  in  technology 
development.  According  to  the  former  project  manager,  the  new  program 
focused  less  on  the  elemental  technology  hurdles  and  more  on  meeting  all 
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user  requirements.  More  expensive  demonstrations  were  necessary  to 
meet  these  broader  requirements  without  necessarily  doing  more  to 
demonstrate  basic  technology  readiness.  It  became  a  more  traditional 
program  with  technology  and  product  development  proceeding  at  the  same 
time,  with  attendant  higher  risks.  In  March  1999,  we  reported  that,  while 
the  ABL  has  made  progress  in  developing  these  technologies,  it  still  faced 
technical  challenges.1 


Other  Incentives  Pressures  exerted  on  weapon  system  programs  can  make  it  advantageous 

to  include  in  their  design  immature  technologies  that  offer  significant 
performance  gains.  One  traditional  source  has  been  the  perceived  threat. 
Users  can  demand  performance  improvements  that  necessitate  the 
application  of  unproven  technologies,  particularly  when  a  fielding  date  is 
mandated,  to  stay  ahead  of  the  threat.  Another  source  is  technologists, 
whether  from  S&T  organizations  or  contractors,  who  see  a  new  weapon 
system  as  an  opportunity  to  apply  a  new  technology.  Also,  the  competition 
for  funds  can  encourage  performance  features — and  requisite 
technologies — that  distinguish  the  new  weapon  system  from  competitors. 

The  F-22  was  justified  as  being  faster,  steaithier,  and  more  lethal  than  other 
fighters,  such  as  the  F-15  and  F-117,  were.  As  a  result,  the  F-22  is  being 
designed  with  several  advanced  technologies,  including  a  very 
sophisticated  suite  of  avionics  that  is  critical  to  its  performance  features 
that  distinguish  it  from  the  other  fighters.  However,  at  the  time  the  F-22 
program  was  launched  in  1986,  the  avionics  technologies  were  immature; 
they  have  since  been  a  source  of  problems  on  the  program.  We  recently 
reported  that  the  development  of  the  F-22’s  integrated  avionics  systems 
continues  to  experience  cost  growth  and  schedule  delays,  more  than 
12  years  into  the  program.2 

A  different  set  of  incentives  causes  leading  commercial  firms  to  make  their 
S&T  organizations  responsible  for  maturing  technologies  to  higher 
readiness  levels.  Commercial  firms  are  aware  of  the  risks  associated  with 
the  high  investment  that  product  development  requires.  They  have  a  strong 
incentive  in  the  realization  that  if  a  product  is  late,  costs  more,  or  performs 


‘Defense  Acouisitions:  DOD  Efforts  to  Develop  Laser  Weapons  for  Theater  Defense  (GAO/NSIAD-99-50, 
Mar.  31, 1999). 

2F-22  Aircraft:  Issues  in  Achieving  Engineering  and  Manufacturing  Development  Costs 
(GAO/NSIAD-99-55,  Mar.  15, 1999). 
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less  than  expected,  the  customer  could  walk  away  from  the  product  and 
the  investment  would  be  lost.  Minimizing  the  possibility  of  technology 
being  the  cause  of  such  problems  is  thus  a  top  priority.  Having  their  S&T 
organizations  reduce  those  risks  is  essential  to  putting  product 
developments  in  the  best  position  to  succeed.  DOD  does  not  have  the  same 
incentives.  DOD  programs  are  not  penalized  if  a  product  is  late,  costs  more, 
or  performs  less  than  expected,  because  the  customer  does  not  walk  away. 


Services  Encouraged 
to  Use  Best  Practices 


Over  the  past  several  years,  DOD  has  encouraged  the  services  to  use  best 
practices  to  streamline  the  current  process  for  acquiring  new  weapon 
systems  in  order  to  make  them  faster,  cheaper,  and  better.  Shorter 
acquisition  cycle  times  are  seen  as  critical  to  making  the  best  use  of 
advances  in  technology.  To  encourage  change,  DOD  has  set  a  goal  to  reduce 
the  average  acquisition  cycle  time  for  all  program  starts  in  fiscal  year  1999 
and  beyond  by  50  percent  over  historical  averages.  DOD  has  several 
initiatives  to  improve  its  technology  development  process  and  to  move 
technologies  to  the  warfighter  faster  and  less  expensively  than  the 
traditional  means.  The  initiatives  also  attempt  to  put  the  organizations  and 
funding  in  place  to  bring  technologies  to  higher  readiness  levels  before  they 
are  included  in  programs.  These  initiatives — defense  technology 
objectives,  advanced  technology  demonstrations,  and  advanced  concept 
technology  demonstrations — call  for  S&T  organizations  to  play  a  bigger 
role  in  managing  technologies  closer  to  the  point  of  product  readiness, 
matching  requirements  to  technology  projects,  and  making  better  use  of 
demonstration  standards. 


Defense  Technology 
Objectives 


Defense  technology  objectives  (DTO)  are  used  to  bring  more  discipline  to 
S&T  projects  and  to  link  them  more  closely  with  weapon  system 
development  programs.  A  DTO  typically  involves  a  particular  technology 
advance,  such  as  high  temperature  materials  for  turbine  engines  and  high 
fidelity  infrared  sensors.  It  can  also  group  several  technologies  into  a  larger 
demonstration.  Each  DTO  identifies  a  specific  technology  advancement 
that  will  be  developed  or  demonstrated,  the  anticipated  date  of  the 
technology  availability,  the  ultimate  customer,  and  the  specific  benefits 
resulting  from  the  technology.  It  places  a  corporate  attention  and 
commitment  on  the  technology  project  by  having  the  technologists, 
product  developer,  and  customer  involved  in  the  project. 

According  to  DOD,  the  focus  of  its  S&T  investment  is  enhanced  and  guided 
through  DTOs.  Each  DTO  must  go  through  a  formal  review  and  approval 
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process  within  DOD  and  must  be  directly  related  to  advancing  the 
operational  concepts  depicted  in  DOD’s  “Joint  Vision  2010”  planning 
document.  According  to  DOD  officials,  those  requirements  have  helped  to 
eliminate  instances  in  which  technologists  work  on  projects  of  particular 
interest  to  them,  but  with  no  military  application,  because  the  projects 
should  be  linked  to  a  specific  warfighter  need.  For  fiscal  year  1999,  DOD 
established  approximately  350  DTOs,  which  accounted  for  $3  billion,  or 
less  than  50  percent,  of  the  funds  DOD  had  allocated  to  S&T  projects.  The 
remaining  funds  were  allocated  to  projects  under  the  jurisdiction  of  each 
military  service  or  other  defense  agencies  and  did  not  go  through  the  same 
review  and  approval  process. 


Advanced  Technology  Advanced  technology  demonstrations  (ATD)  are  intended  to  more  rapidly 

Demonstrations  evolve  and  demonstrate  new  technologies  so  they  can  be  incorporated  into 

a  product,  if  warranted.  An  ATD  has  four  characteristics  that  distinguish  it 
from  a  conventional  S&T  project.  They  (1)  require  large-scale  resources; 
(2)  involve  the  user;  (3)  use  specific  cost,  schedule,  and  performance 
metrics;  and  (4)  identify  a  target  product  for  inclusion.  An  ATD  is  managed 
by  an  S&T  organization  and  should  conclude  with  an  operational 
demonstration  of  the  potential  capabilities  of  the  technology,  equating  to  a 
TRL  5  or  6.  The  original  approach  to  the  ABL  was  essentially  an  ATD 
approach.  Most  ATDs  use  laboratory  hardware  to  demonstrate  the 
potential  capability  of  nonproduct  specific  technologies  and  not  prototype 
hardware.  If  the  technology  is  determined  to  be  feasible  and  provides  some 
military  use,  then  it  may  proceed  to  the  program  definition  and  risk 
reduction  phase  of  an  acquisition  program.  From  that  point,  the  product 
developer  completes  the  technology  development  for  a  specific  product. 


Advanced  Concept  In  1994,  DOD  initiated  Advanced  Concept  Technology  Demonstrations 

Technology  Demonstrations  (ACTD)  to  help  expedite  the  transition  of  mature  technologies  from  the 

developers  to  the  warfighters.  ACTDs  are  intended  to  help  the  DOD 
acquisition  process  adapt  to  budget  constraints  while  developing 
technology  more  rapidly.  The  purpose  of  an  ACTD  is  to  assess  the  military 
use  of  a  capability,  such  as  a  weapon,  comprised  of  mature  technologies. 
Typically,  ACTDs  last  2  to  4  years  and  consist  of  building  and 
demonstrating  a  prototype  to  provide  a  warfighter  the  opportunity  to 
assess  a  prototype’s  capability  in  realistic  operational  scenarios.  From  this 
demonstration,  the  warfighter  can  refine  operational  requirements,  develop 
an  initial  concept  of  operation,  and  determine  the  military  use  of  the 
technology  before  it  proceeds  to  the  product  development  process. 
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According  to  DOD,  ACTDs,  which  are  managed  by  S&T  organizations,  will 
be  a  key  mechanism  to  ensure  technology  development  is  separated  from 
product  development.  In  related  work  on  unmanned  air  vehicles,  we  found 
that  ACTDs  provided  decisionmakers  credible  data  that  they  used  to 
terminate  efforts  or  transition  the  demonstrator  to  an  acquisition  program. 
In  these  cases,  ACTDs  put  decisionmakers  in  a  better  position  to  be 
gatekeepers.  However,  we  have  reported  that  the  ACTD  program  needs  to 
be  improved.3  We  found  that  DOD’s  process  for  selecting  program 
candidates  does  not  include  adequate  criteria  for  assessing  the  maturity  of 
proposed  technology  and  has  resulted  in  the  approval  of  projects  that 
included  immature  technologies.  We  found  that  the  use  of  specific  criteria 
for  determining  maturity  was  a  best  practice  in  the  most  successful 
technology  development  cases  we  examined. 


Two  Unique  DOD 
Projects  May  Provide 
Lessons  on  How  to 
Enable  S&T 
Organizations  to 
Manage  Technology 
Further 


Two  DOD  projects  are  using  S&T  organizations  to  manage  technology 
development  to  higher  readiness  levels.  One,  the  Army’s  Future  Scout  and 
Cavalry  System,  is  using  a  modified  ATD  to  mature  technologies  and  make 
performance  trade-offs  in  the  more  flexible  environment  provided  by 
S&T.  The  other  is  a  joint  government  and  industry  program,  which  the  Air 
Force  Research  Laboratory  is  managing  to  reduce  the  risks  associated  with 
new  jet  engine  technologies.  These  projects  may  provide  insights  on  how 
S&T  organizations  could  routinely  play  a  bigger  role  in  maturing 
technologies  enough  for  safe  inclusion  on  weapon  system  programs.  They 
may  also  clarify  the  concern  that  playing  a  bigger  role  in  technology 
maturation  could  cause  S&T  organizations  to  do  less  basic  research  and 
technology  development. 


Future  Scout  and  Cavalry 
System 


In  fiscal  year  1997,  the  Army  began  piloting  a  variation  of  an  ATD  that  is 
designed  to  help  bridge  the  gap  between  technology  development  and 
product  development  by  expanding  the  S&T  community’s  role  in  managing 
technologies  further  into  the  development  cycle.  The  Army’s  initiative, 
called  Fast  Track,  is  intended  to  reduce  cost  and  cycle  time  by  bypassing 
the  program  definition  and  risk  reduction  phase  of  the  DOD  acquisition 
process.  The  Army  is  testing  this  concept  with  its  Future  Scout  and  Cavalry 
System  project.  In  this  project,  the  Army  will  design,  develop,  and  build  a 
demonstrator  vehicle  to  show  the  technical  feasibility  of  the  weapon.  All  of 


3Defense  Acquisition:  Advanced  Concept  Technology  Demonstration  Program  Can  Be  Improved 
(GAO/NSIAD-99-4,  Oct  16, 1998). 
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these  tasks  will  be  done  under  the  management  of  the  Army’s  S&T 
organization.  The  Army  believes  that  a  more  extensive  S&T  project  will 
make  the  program  definition  phase  unnecessary  and  estimates  that  this 
concept  will  reduce  the  development  process  by  as  much  as  4  years  and 
save  about  $400  million.  We  did  not  review  the  Future  Scout  project  in 
terms  of  its  affordability,  feasibility,  or  any  impacts  it  may  have  on  the 
Army’s  S&T  budget.  Figure  4.2  compares  the  Fast  Track  development 
process  with  the  traditional  approach. 


Figure  4.2:  Comparison  of  Traditional  Technology  Development  Process  With  the  Army’s  Fast  Track  Approach 
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While  we  do  not  necessarily  agree  that  the  first  phase  of  the  acquisition 
cycle  can  be  omitted,  so  far  the  Future  Scout  project  is  emulating 
technology  development  practices  like  those  we  observed  in  the  successful 
cases.  First,  it  has  established  demonstration  criteria  that  must  be  met 
before  the  technology  enters  product  development.  Second,  it  has  also 
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established  forums  that  involve  key  players  on  the  technology  path  to  keep 
them  informed  of  the  technology’s  development  progress.  For  example,  the 
acquisition  program  manager  will  be  integrated  into  the  development 
project  during  the  final  1.5  years  of  the  S&T  program.  This  should  provide  a 
good  link  between  the  technology  development  and  product  development, 
allowing  the  program  manager  to  fully  understand  the  technology  before 
product  development  begins.  Finally,  by  allowing  an  S&T  organization  the 
flexibility  to  manage  technologies  further  into  the  development  cycle, 
Army  officials  believe  they  will  be  able  to  make  trade-offs  among  cost, 
schedule,  and  performance  requirements  before  program  launch,  without 
raising  concerns  about  the  state  of  the  project  or  breaching  baselines  that 
had  been  set  without  enough  knowledge. 

While  this  concept  comes  closer  to  the  most  successful  technology 
development  cases  we  reviewed,  it  still  embodies  greater  technical  risk. 
The  Army  expects  to  demonstrate  some  performance  capabilities  of  the 
vehicle  before  the  product  development  phase  begins.  However,  the 
demonstrator  vehicle  will  only  be  about  75  percent  of  a  complete 
prototype,  which  means  some  key  technologies  will  not  be  demonstrated 
to  high  readiness  levels  before  that  phase  begins.  Nonetheless,  the  project 
manager  equated  the  expected  overall  technical  maturity  of  the  vehicle  at 
transition  to  a  TRL  6.  The  Army  considers  this  a  medium  or  acceptable 
level  of  risk,  and  it  is  willing  to  enter  product  development  with  some 
immature  technologies.  If,  however,  product  development  begins  at 
engineering  and  manufacturing  development,  this  risk  could  be  assessed  as 
high,  based  on  TRLs.  Figure  4.3  shows  an  artist’s  concept  of  the  Future 
Scout  and  Cavalry  System. 
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Figure  4.3:  Future  Scout  and  Cavalry  System 


An  Army  S&T  organization  is  maturing  technologies  and  proposing  performance  trade-offs  for  the 
Future  Scout  and  Cavalry  System  before  program  launch 

Source:  DOD. 


Integrated  High  The  Integrated  High  Performance  Turbine  Engine  Technology  program— a 

Performance  Turbine  joint  government  and  industry  effort — is  focused  on  developing 

Engine  Technology  Program  technol°gies  for  more  affordable  and  higher  performance  turbine  engines 
°  for  both  missiles  and  aircraft.  It  is  a  technology  validation  program  and  is 

managed  by  an  S&T  organization  to  perform  demonstrations  of  various 
engine  technologies  to  higher  readiness  levels  than  most  S&T  projects. 
After  the  demonstrations,  the  technologies  enter  a  product  development 
program.  The  program  takes  the  technology  through  a  series  of  tests  that 
range  from  individual  component  tests  to  full-scale  engine  demonstrations. 
The  program  has  established  strong  links  with  the  acquisition  programs  for 
which  the  technologies  are  intended.  For  example,  Air  Force  Research 
Laboratory  officials  informed  us  that  they  established  formal  technology 
transition  plans  with  the  F-22  and  Joint  Strike  Fighter  programs  that 
document  agreements  on  what  technology  development  activities  will  be 
performed  to  support  the  programs.  Representatives  from  each  program 
office  are  invited  to  all  technology  demonstrations  and  are  kept  informed 
about  demonstrated  progress. 
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As  part  of  the  program,  the  Air  Force  developed  a  set  of  standards  to  assess 
the  readiness  levels  of  technologies  similar  to  NASA’s  TRLs.  According  to 
the  Air  Force,  the  S&T  organization  uses  these  standards  to  determine 
when  the  project  has  been  completed.  These  standards  were  the  first 
application  of  readiness  levels  by  the  Air  Force  Research  Laboratory.  There 
are  five  technology  readiness  levels  ranging  from  component-level  tests  in 
a  laboratory  to  the  highest  level  involving  actual  flight  tests  of  engines.  The 
program  typically  does  not  take  technologies  to  the  highest  readiness  level 
(flight  test)  because  of  the  high  cost.  The  program  stops  when  it  has  been 
determined  the  technology  is  well  defined  within  acceptable  boundaries 
and  a  good  correlation  exists  between  test  results  and  engineering 
predictions.  This  readiness  level  would  translate  to  a  TRL  of  5  or  6,  as  used 
in  this  report.  The  final  step  of  the  technology  development  is  left  to  the 
product  developer  who  determines  if  the  technology  can  be  packaged  and 
integrated  into  the  final  product. 
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Conclusions  Clearly,  DOD’s  continued  advancement  of  new  technologies  is  essential  to 

the  continued  superiority  of  its  weaponry.  The  leading  edge  military 
capabilities  the  United  States  possesses  today,  such  as  stealth  aircraft, 
precision  munitions,  and  intelligence-gathering  satellites,  bear  witness  to 
the  effects  of  such  technical  advances.  At  the  same  time,  the  incorporation 
of  advanced  technologies  before  they  are  mature  has  been  a  major  source 
of  cost  increases,  schedule  delays,  and  performance  problems  on  weapon 
systems.  As  DOD  contemplates  increasing  its  annual  investment  in  new 
weapons  to  $60  billion,  the  expectations  on  program  managers  are  great: 
they  must  develop  and  field  weapons  of  superior  capability  more  quickly 
and  less  expensively  than  in  the  past.  The  way  advanced  technologies  are 
matured  and  included  in  weapon  systems  will  play  a  central  role  in  meeting 
these  expectations.  Although  different  ways  to  better  assimilate  new 
technologies  into  weapons  are  legitimate  topics  for  debate,  that  it  has  to  be 
done  better  is  not. 

The  leading  commercial  firms’  practices  have  produced  results  that 
resemble  those  sought  by  DOD:  more  technically  advanced,  higher  quality 
products,  developed  in  significantly  less  time,  and  less  expensively  than 
their  predecessors.  Managing  the  development  of  advanced  technology 
differently-and  separately-from  the  development  of  a  product  has  been 
key  to  these  results.  The  firms  insist  that  advanced  technology  reach  a  high 
level  of  maturity,  the  point  at  which  the  knowledge  about  that  technology  is 
essentially  complete,  before  allowing  it  into  a  product  development.  By 
separating  the  two,  the  firms  lessen  the  product  manager’s  burden  and 
place  that  person  in  a  better  position  to  succeed  in  delivering  the  product. 
These  practices  may  not  necessarily  accelerate  the  pace  at  which 
technology  matures.  In  fact,  several  of  the  commercial  technologies  we 
reviewed  took  10  years  or  more  to  get  to  market.  The  clear  beneficiaries  of 
the  practices  are  the  product  developments,  for  which  the  investments  are 
much  larger,  and  time  translates  into  significantly  more  resources  than  in  a 
technology  project.  Adapting  these  practices  on  its  weapon  system 
programs  can  help  DOD  to  reduce  costs  and  the  time  from  product  launch 
to  fielding,  and  use  technology  advances  as  they  become  available  more 
frequently. 

Separating  technology  development  from  product  development  calls  for  a 
new  approach  to  managing  technology  development.  Two  conditions  are 
essential  to  such  an  approach.  First,  the  right  environment  for  maturing 
technologies  must  exist.  A  practice  that  is  instrumental  in  providing  this 
environment  is  maturing  technology  to  achieve  product  readiness  before  it 
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is  constrained  by  the  rules  of  an  acquisition  program.  In  the  successful 
DOD  cases  we  reviewed,  this  environment  was  provided  by  S&T 
organizations  or  a  team  of  S&T  and  product  developers  who  managed 
technologies  to  high  readiness  levels  before  they  were  included  in  an 
acquisition  program.  These  organizations  provided  an  environment  more 
conducive  to  the  ups  and  downs  normally  associated  with  the  discovery 
process.  A  corollary  practice  is  agreeing  on  what  level  of  knowledge  is 
needed  about  a  new  technology  before  it  is  considered  for  inclusion  in  a 
product  design.  When  that  knowledge  level  does  not  exist,  the  flexibility  for 
S&T  organizations  and  product  managers  to  either  take  the  time  to  mature 
the  technology  or  trade  off  product  requirements  until  they  can  be  met  with 
mature  technology  is  essential.  It  is  a  rare  program  that  can  proceed  with  a 
gap  between  product  requirements  and  maturity  of  key  technologies  and 
still  be  delivered  on  time  and  within  costs.  Second,  S&T  and  product 
managers  must  be  provided  with  the  disciplined  processes,  information, 
standards,  and  authority  to  make  good  handoffs  of  technology  to  product. 
Prepared  with  the  tools  and  authority  to  make  sound  handoff  decisions, 
both  S&T  and  product  managers  can  function  as  gatekeepers  to  safeguard 
the  product  development  from  undue  technology  risks. 

Leading  commercial  firms  have  adopted  this  approach  as  a  matter  of 
necessity  and  have  used  the  organizations,  tools,  and  other  practices  to 
foster  technology  development  and  improve  the  outcomes  of  product 
developments.  The  high  stakes  stemming  from  the  large  investment 
required  for  a  new  product  and  the  risks  if  the  product  does  not  meet 
customer  needs  reinforce  this  approach  in  leading  commercial  firms.  The 
DOD  cases  that  followed  a  similar  approach  were  realizing  better  program 
outcomes,  at  least  in  the  sense  that  the  programs  avoided  key  technology 
development  problems.  Yet,  these  cases  are  not  the  norm  for  DOD 
programs  for  several  reasons. 

•  More  typically,  the  commitment  to  develop  and  produce  a  weapon 
system  is  made  before  a  match  between  technology  and  weapon  system 
requirements  exists. 

•  DOD  programs  operate  under  different  conditions  that  make  it  more 
difficult — and  less  rewarding — to  separate  technology  development 
from  product  development. 

•  Budget  realities  make  it  more  difficult  for  S&T  organizations  to  carry 
technologies  to  the  high  readiness  levels  needed  to  meet  product 
requirements;  such  resources  are  more  available  within  product 
developments. 
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•  The  pressures  to  show  the  unequalled  performance  necessary  to  win 
funding  encourage  including  promising,  but  immature,  technologies  in 
weapon  system  designs. 

It  will  take  procedural,  organizational,  and  cultural  changes  within  DOD’s 
acquisition  process  to  foster  an  environment  in  which  (1)  technologies  can 
be  successfully  matured  outside  the  purview  of  weapon  system  programs, 
(2)  programs  can  be  relieved  of  the  pressures  to  include  immature 
technologies  and  the  unrealistic  expectations  that  the  technologies  will 
conform  to  tight  cost  and  schedule  projections,  and  (3)  technology 
advances  will  not  stall  due  to  inadequate  funding  or  lack  of  identification 
with  a  product  in  the  later,  more  expensive  stages  of  demonstration. 

Experience  has  shown  that  such  an  approach  can  work  within  DOD  on 
individual  cases.  DARPA  played  a  primary  role  in  managing  the  transition 
of  the  nonpenetrating  photonics  mast  technology  to  the  Virginia  class 
attack  submarine.  The  Integrated  High  Performance  Turbine  Engine 
Technology  program  has  carried  advanced  jet  engine  technologies  to  TRLs 
of  between  5  and  6  for  successful  inclusion  into  programs.  In  the  Future 
Scout  program,  an  Army  S&T  organization,  augmented  by  product 
development  staff,  is  managing  an  ATD  to  lower  the  risk  of  key 
technologies  before  a  product  development  program  is  launched.  However, 
it  remains  to  be  seen  whether  the  Army  will  be  successful  in  using  large  and 
expensive  S&T  projects,  such  as  the  Future  Scout  program,  without 
affecting  other  Army  S&T  projects.  A  challenge  for  DOD  will  be  whether 
the  lessons  learned  from  these  individual  cases  offer  an  approach  that  has 
DOD-wide  application.  Meeting  this  challenge  is  essential  to  fielding 
technologically  superior  weapons  more  quickly  and  within  predicted  costs. 


Recommendations  We  have  previously  recommended  that  DOD  separate  technology 

development  from  weapon  system  programs.  That  recommendation  was 
made  without  prejudice  toward  the  necessity  of  technology  development 
but  rather  with  the  intent  that  programs  could  be  better  managed  if  such 
development  was  conducted  outside  of  a  program  manager’s  purview. 
Similarly,  the  recommendations  that  follow  are  made  without  prejudice 
toward— or  the  intention  of  compromising — the  basic  research  and  other 
activities  that  S&T  organizations  perform.  We  recognize  that 
implementation  of  these  recommendations  will  have  organizational, 
funding,  and  process  implications  and  will  require  the  cooperation  of  the 
Congress. 


Page  63 


GAO/NS IAD-9 9-1 62  Best  Practices 


Chapter  5 

Conclusions  and  Recommendations 


To  help  ensure  that  new  technologies  are  vigorously  pursued  and 
successfully  moved  into  weapon  system  programs,  we  recommend  that  the 
Secretary  of  Defense  adopt  a  disciplined  and  knowledge-based  method  for 
assessing  technology  maturity,  such  as  TRLs,  DOD-wide.  This  practice 
should  employ  standards  for  assessing  risks  of  handoff  to  program 
managers  that  are  based  on  a  technology’s  level  of  demonstration  and  its 
criticality  to  meeting  the  weapon  system’s  requirements. 

With  these  tools  in  hand,  we  recommend  that  the  Secretary  (1)  establish 
the  place  at  which  a  match  is  achieved  between  key  technologies  and 
weapon  system  requirements  as  the  proper  time  for  committing  to  the  cost, 
schedule,  and  performance  baseline  for  developing  and  producing  that 
weapon  system  and  (2)  require  that  key  technologies  reach  a  high  maturity 
level — analogous  to  TRL  7 — before  making  that  commitment.  This  would 
approximate  the  launch  point  for  product  development  as  practiced  by 
leading  commercial  firms. 

We  recommend  that  the  Secretary  find  ways  to  ensure  that  the  managers 
responsible  for  maturing  the  technologies  and  designing  weapon  systems 
before  product  development  are  provided  the  more  flexible  environment 
that  is  suitable  for  the  discovery  of  knowledge,  as  distinct  from  the  delivery 
of  a  product.  Providing  more  flexibility  will  require  the  cooperation  of 
requirements  managers  and  resource  managers  so  that  rigid  requirements 
or  the  threat  of  jeopardizing  the  funding  planned  to  start  product 
development  will  not  put  pressure  on  program  managers  to  accept 
immature  technologies.  Such  an  environment  may  not  be  feasible  if  the 
program  definition  and  risk  reduction  phase  remains  the  effective  launch 
point  for  an  entire  weapon  system  program. 

An  implication  of  these  recommendations  is  that  S&T  organizations  will 
have  to  play  a  greater  role  in  maturing  technologies  to  higher  levels  and 
should  be  funded  accordingly.  Therefore,  we  recommend  that  the  Secretary 
of  Defense  evaluate  the  different  ways  S&T  organizations  can  play  a  greater 
role  in  helping  technologies  reach  high  levels  of  maturity  before  product 
development  begins.  For  example,  given  that  a  technology  has  sufficient 
potential  for  application  to  a  weapon  system,  at  a  minimum,  an  S&T 
organization  should  be  responsible  for  taking  a  technology  to  TRL  6  before 
it  is  handed  off  to  a  program  office  at  the  program  definition  and  risk 
reduction  phase.  During  this  phase,  the  program  manager  would  be 
responsible  for  maturing  the  technology  to  TRL  7  before  it  is  included  in  an 
engineering  and  manufacturing  development  program.  In  a  situation  where 
a  single,  design-pacing  technology  is  to  be  developed  for  a  known 
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application — like  the  nonpenetrating  periscope — an  S&T  organization 
should  be  required  to  mature  that  technology  to  TRL  7  before  it  is  turned 
over  to  a  product  development  manager.  S&T  organizations  could  play  a 
similar  role  when  a  significant  new  technology  is  being  prepared  for 
insertion  into  an  existing  weapon  system.  Finally,  when  multiple  new 
technologies  are  to  be  merged  to  create  a  weapon  system,  S&T 
organizations  should  be  required  to  bring  key  technologies  to  TRL  6  and 
then  become  part  of  a  hybrid  organization  with  product  developers  to 
integrate  the  technologies  and  bring  them  to  TRL  7  before  handing  full 
responsibility  to  a  product  development  manager. 

To  help  guard  against  the  possibility  that  the  more  basic  research  and 
technology  development  activities  would  be  compromised  by  having  S&T 
organizations  routinely  take  key  technologies  to  TRL  6  or  higher,  we 
recommend  that  the  Secretary  extract  lessons  from  the  nonpenetrating 
periscope,  the  AAAV,  and  the  Army’s  Future  Scout  programs,  and  other 
ATD  and  ACTD  programs.  Specifically,  the  Secretary  should  assess  whether 
the  resources  needed  to  enable  S&T  organizations  to  play  a  leading  role  in 
the  development  of  technologies  and,  in  some  cases,  preliminary  system 
design,  detracted  from  or  displaced  more  basic  research  and  technology 
development  programs. 

Finally,  we  recommend  that  the  Secretary  empower  managers  of  product 
development  programs  to  refuse  to  accept  key  technologies  with  low  levels 
of  demonstrated  maturity.  The  Secretary  can  encourage  this  behavior 
through  supportive  decisions  on  individual  programs,  such  as  by  denying 
proposals  to  defer  the  development  of  key  technologies  and  by  favoring 
proposals  to  lengthen  schedules  or  lessen  requirements  to  reduce 
technological  risk  early. 


Agency  Comments  and 
Our  Evaluation 


DOD  generally  concurred  with  a  draft  of  this  report  and  its 
recommendations,  noting  that  the  traditional  path  to  new  weapon  system 
development  is  no  longer  affordable  or  necessary  (see  app.  I).  DOD  stated 
that  it  has  embarked  upon  a  “Revolution  in  Business  Affairs”  that  will 
enable  new  technologies  to  be  developed  more  efficiently  and  effectively.  It 
believes  that  the  first  steps  in  this  direction  have  already  been  taken  but 
agrees  that  more  progress  needs  to  be  made.  DOD  agreed  that  TRLs  are 
necessary  in  assisting  decisionmakers  in  deciding  on  when  and  where  to 
insert  new  technologies  into  weapon  system  programs  and  that  weapon 
system  managers  should  ensure  that  technology  is  matured  to  a  TRL  7 
before  insertion  occurs.  DOD  concurred  that  S&T  organizations  should  be 
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involved  in  maturing  technologies  to  high  levels,  such  as  TRL  6,  before 
transitioning  to  the  engineering  and  manufacturing  development  phase  and 
agreed  to  assess  the  impact  of  this  involvement  on  other  S&T  resources. 
We  note  that  the  best  practice  is  to  mature  technology  to  at  least  a  TRL  7 
before  starting  the  engineering  and  manufacturing  development  phase, 
whether  the  technology  is  managed  by  an  S&T  organization,  a  weapon 
system  program  manager,  or  a  hybrid  of  the  two  organizations. 

DOD  noted  that  while  TRLs  are  important  and  necessary,  the  increasing 
projected  life  for  new  weapon  systems,  total  ownership  costs,  and  urgency 
based  upon  threat  assessments  are  also  important  considerations  for 
system  development  decisions.  We  agree  and  note  that  our 
recommendations  are  not  intended  to  cover  all  aspects  of  weapon  system 
development  decisions  or  to  suggest  that  technology  maturity  is  the  only 
factor  in  such  decisions.  Rather,  the  recommendations  are  in  keeping  with 
the  purpose  of  the  report,  “to  determine  whether  best  practices  offer 
methods  to  improve  the  way  DOD  matures  new  technology  so  that  it  can  be 
assimilated  into  weapon  system  programs  with  less  disruption.”  We  believe 
that  a  knowledge-based  approach  to  maturing  technology,  such  as  TRLs, 
can  benefit  other  considerations  as  well.  For  example,  decisions  on  what 
technologies  to  include  in  a  weapon  system  and  when  to  include  them  can 
have  a  significant  bearing  on  its  total  ownership  costs. 

DOD  stated  that  there  should  be  an  established  point  for  the  transition  of 
technologies  and  that  it  plans  to  supplement  its  milestone  review  process 
with  additional  guidance  in  the  next  revisions  to  DOD  Directive  5000.2R.  It 
also  stated  that  its  policy  on  the  evolutionary  approach  to  weapon 
acquisitions  should  be  developed  in  consonance  with  the  technology 
transition  strategy.  We  cannot  comment  on  the  revisions  to  the  directive  or 
the  evolutionary  acquisition  policy  because  they  have  yet  to  be  published. 
However,  under  the  current  milestone  review  process,  the  pressures  placed 
on  a  program  during  the  program  definition  and  risk  reduction  phase — 
when  much  technology  development  occurs — can  operate  against  the 
flexibility  and  judgments  that  are  needed  to  mature  technologies.  If  the 
revisions  to  the  directive  supplement  the  current  milestones  without 
relieving  the  pressures  brought  to  bear  on  programs  as  they  are  launched  in 
the  program  definition  and  risk  reduction  phase,  it  will  remain  difficult  to 
discourage  the  acceptance  of  immature  technologies  in  the  design  of  new 
weapon  systems.  To  relieve  these  pressures,  we  encourage  DOD,  as  it 
develops  the  directive  and  the  evolutionary  acquisition  policy,  to  separate 
technology  development  from  product  development  and  to  redefine  the 
launch  point  for  a  program  as  the  point  at  which  enough  knowledge  has 
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been  gained  to  ensure  that  a  match  is  reached  between  the  maturity  of  key 
technologies  and  weapon  system  requirements. 

DOD  also  stated  that  program  managers  already  have  the  ability  to  reject 
inappropriately  mature  technologies,  and  to  the  extent  technology 
immaturity  affects  acquisition  baselines,  to  advise  acquisition  executives  of 
feasible  alternatives.  We  did  not  find  this  to  be  the  case  in  our  review. 
Rather,  we  found  that  the  program  managers’  ability  to  reject  immature 
technologies  is  hampered  by  (1)  untradable  requirements  that  force 
acceptance  of  technologies  despite  their  immaturity  and  (2)  reliance  on 
tools  forjudging  technology  maturity  that  fail  to  alert  the  managers  of  the 
high  risks  that  would  prompt  such  a  rejection.  As  noted  in  the  report,  once 
a  weapon  system  program  begins,  the  environment  becomes  inflexible  and 
deviations  to  program  baselines  can  attract  unwanted  attention.  This 
reality  limits  the  program  managers’  ability  to  reject  immature 
technologies. 
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Technology  readiness  level 

Description 

1.  Basic  principles  observed  and  reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be  translated  into 
applied  research  and  development.  Examples  might  include  paper  studies  of  a 
technology’s  basic  properties. 

2.  Technology  concept  and/or  application 
formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can  be 
invented.  The  application  is  speculative  and  there  is  no  proof  or  detailed  analysis  to 
support  the  assumption.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  and  experimental  critical  function 
and/or  characteristic  proof  of  concept. 

Active  research  and  development  is  initiated.  This  includes  analytical  studies  and 
laboratory  studies  to  physically  validate  analytical  predictions  of  separate  elements  of 
the  technology.  Examples  include  components  that  are  not  yet  integrated  or 
representative. 

4.  Component  and/or  breadboard  validation  in 
laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that  the  pieces  will  work 
together.  This  is  relatively  “low  fidelity”  compared  to  the  eventual  system.  Examples 
include  integration  of  “ad  hoc”  hardware  in  a  laboratory. 

5.  Component  and/or  breadboard  validation  in 
relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  that  the 
technology  can  be  tested  in  a  simulated  environment.  Examples  include  “high  fidelity” 
laboratory  integration  of  components. 

6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  the  breadboard 
tested  for  TRL  5,  is  tested  in  a  relevant  environment.  Represents  a  major  step  up  in  a 
technology’s  demonstrated  readiness.  Examples  include  testing  a  prototype  in  a  high 
fidelity  laboratory  environment  or  in  simulated  operational  environment. 

7.  System  prototype  demonstration  in  an 
operational  environment. 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up  from 

TRL  6,  requiring  the  demonstration  of  an  actual  system  prototype  in  an  operational 
environment,  such  as  in  an  aircraft,  vehicle  or  space.  Examples  include  testing  the 
prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and  “flight  qualified” 
through  test  and  demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions. 

In  almost  all  cases,  this  TRL  represents  the  end  of  true  system  development. 

Examples  include  developmental  test  and  evaluation  of  the  system  in  its  intended 
weapon  system  to  determine  if  it  meets  design  specifications. 

9.  Actual  system  “flight  proven”  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions, 
such  as  those  encountered  in  operational  test  and  evaluation.  In  almost  all  cases,  this 
is  the  end  of  the  last  “bug  fixing”  aspects  of  true  system  development.  Examples 
include  using  the  system  under  operational  mission  conditions. 
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ACQUISITION  AND 
TECHNOLOGY 


THE  UNDER  SECRETARY  OF  OE r  i "N  - 

3010  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301  30  JO 


JUL  I  6  1999 


Mr.  Louis  J.  Rodrigues 
Director,  Defense  Acquisition  Issues 
National  Security  and  International 
Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 


Dear  Mr.  Rodrigues: 

This  is  the  Department  of  Defense  response  to  the  General 
Accounting  Office  (GAO)  draft  report,  "BEST  PRACTICES:  Managing 
Advanced  Technology  Development  Can  Improve  Weapon  System 
Outcomes,"  dated  June  7,  1999  (GAO  Code  707336/OSD  Case  1836). 

The  Department  concurs  with  the  GAO  draft  report. 

The  Department  agrees  with  the  GAO  that  much  of  the 
traditional  path  to  new  weapon  system  development  is  no  longer 
affordable  or  necessary.  Following  the  lead  of  commercial  best 
practices,  the  Department  has  embarked  upon  a  Revolution  in 
Business  Affairs  that  will  enable  the  acquisition  of  new 
technologies  more  efficiently  and  effectively.  The  first  steps 
in  this  new  direction  have  already  been  taken,  but,  as  GAO  points 
out,  more  progress  needs  to  be  made.  The  Department  welcomes  the 
GAO's  recommendations,  and  our  specific  responses  to  these 
recommendations  are  included  herein  as  an  attachment. 

Thank  you  for  the  opportunity  to  review  and  comment  on  the 
report.  Technical  corrections  to  the  report  were  separately 
provided.  The  professionalism  and  the  level  of  cooperation 
between  my  staff  and  yours  are  greatly  appreciated,  and  we  look 
forward  to  working  with  your  staff  again  in  the  future. 

Sincerely, 


Attachment : 
As  stated 
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Now  on  pp.  7  and  63-64. 


Now  on  pp.  7  and  63-64. 


Now  on  pp.  7  and  63-64. 


GENERAL  ACCOUNTING  OFFICE  DRAFT  REPORT  -  DATED  JUNE  7,  1999 
(GAO  CODE  707336)  OSD  CASE  1836 


"BEST  PRACTICES  s  MANAGING  ADVANCED  TECHNOLOGY  DEVELOPMENT 
CAN  IMPROVE  WEAPON  SYSTEM  OUTCOMES" 


DOD  RESPONSE 

RECOMMENDATION  1:  To  help  ensure  that  new  technologies  are  vigorously  pursued 
yet  successfully  moved  into  weapon  system  programs,  the  GAO  recommended  that 
the  Secretary  of  Defense  adopt  a  disciplined  and  knowledge-based  practice  for 
assessing  technology  maturity,  such  as  technology  readiness  levels  (TRLS) , 
DOD-wide .  The  GAO  asserted  that  this  practice  should  employ  standards  that 
are  based  on  a  technology's  level  of  demonstration  and  its  criticality  to 
meeting  the  weapon's  requirements,  (p.  5,  p. 4 5 /GAO  Draft  Report) 

DOD  RESPONSE:  Partially  concur.  The  Department  agrees  that  TRLS  are  an 
important  input  and  are  necessary,  but  adds  that  they  are  not  sufficient  alone 
to  decide  when  and  where  to  insert  new  technologies  into  weapon  system 
programs.  Military  system  development  decisions  require  a  total  ownership 
cost  approach  through  the  entire  life  cycle  of  a  system,  not  just  an 
assessment  of  technology  maturity.  Given  the  increasing  projected  life  for 
new  military  systems,  additional  considerations  also  present  include  the 
training  and  logistics  costs  over  time,  coupled  with  urgency  based  upon  threat 
assessments. 

RECOMMENDATION  2:  With  those  tools  in  hand,  the  GAO  recommends  that  the 
Secretary  of  Defense  (1)  establish  the  point  at  which  a  match  is  achieved 
between  key  technologies  and  weapon  system  requirements  as  the  proper  point 
for  committing  to  the  cost,  schedule,  and  performance  baseline  for  developing 
and  producing  that  weapon  system,  and  (2)  require  that  key  technologies  reach 
a  maturity  level-analogous  to  TRL  7-before  making  that  commitment.  The  GAO 
indicated  that  this  would  be  analogous  to  the  launch  point  for  product 
development  as  practiced  by  leading  firms,  (p.  5,  p. 4 5 /GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  The  Department  agrees  with  the  GAO  that  there  should 
be  an  established  point  for  transition  and  that  decision  points  as  currently 
identified  in  the  Milestone  review  process  will  be  supplemented  with 
additional  guidance  in  the  next  revisions  to  the  DoDD  5000. 2R.  The  Department 
believes  that  the  evolutionary  acquisition  strategy  for  new  weapons  systems 
should  be  developed  in  consonance  with  the  technology  transition  or  insertion 
strategy  and  additional  guidance  will  be  developed  for  the  next  revisions  to 
the  DoDD  5000. 2R. 

RECOMMENDATION  3:  The  GAO  also  recommended  that  the  Secretary  find  ways  to 
ensure  that  the  managers  responsible  for  maturing  the  technologies  and 
designing  weapons  systems  before  product  development  begins  are  provided  the 
more  flexible  environment  that  is  suitable  for  the  discovery  of  knowledge,  as 
distinct  from  the  delivery  of  product.  The  GAO  suggested  that  providing  more 
flexibility  will  take  the  cooperation  of  requirements  managers  and  resource 
managers  so  that  rigid  requirements  or  the  threat  of  jeopardizing  the  funding 
planned  to  start  product  development  would  not  put  pressure  on  managers  to 
accept  immature  technologies.  According  to  the  GAO,  such  an  environment  may 
not  be  feasible  if  the  program  definition  and  risk  reduction  phase  remains  the 
effective  launch  point  for  an  entire  weapon  system  program,  (p.  5,  p. 45/GAO 
Draft  Report) 
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Now  on  pp.  7  and  63-64. 


POD  RESPONSE:  Partially  Concur.  DoD  assigns  responsibility  to  different 
individuals  for  research  projects  related  to  technology  and  to  major  systems 
development,  and  the  individuals  are  frequently  in  different  organizations, 
such  as  DARPA  or  a  laboratory  for  research  programs,  and  a  systems  command  for 
an  acquisition  program.  Managers  of  research  projects  generally  enjoy  a  more 
flexible  environment  than  program  managers  do  who  are  charged  with  developing 
systems  to  meet  operational  capability  dates  within  constrained  budgets  as 
well  as  within  constraints  such  as  interoperability  and  logistics 
compatibility.  The  Department's  technology  managers  already  promote  and  offer 
technology  opportunities  to  their  war fighting  and  weapon  system  program 
managers  by  delivering  products  to  meet  Warfighter  needs  within  life  cycle 
cost,  program  schedule  and  performance  constraints.  We  agree  that  the 
Department  must  caution  against  the  singular  notion  that  technology 
opportunity,  or  advance,  is  more  important  than  these  last  factors.  The  S&T 
community  is  closely  coupled  with  the  Department's  program  managers  to  offer 
enhanced  capability  options-throughout  the  life  cycle-from  the  DOD's  Advanced 
Technology  Demonstrations  (6.3)  and  supporting  Applied  Research  (6.2) 
programs.  Constraints  on  technology  often  must  be  applied  so  we  can  develop 
the  truly  essential  and  militarily  relevant  technologies  within  our  budget. 

RECOMMENDATION  4 :  The  GAO  indicated  that  an  implication  of  these 
recommendations  is  that  Science  and  Technology  (S&T)  organizations  will  have 
to  play  an  greater  role  in  maturing  technologies  to  that  level  and  should  be 
funded  accordingly.  Therefore,  the  GAO  recommended  that  the  Secretary  of 
Defense  evaluate  the  different  ways  S&T  organizations  can  play  a  greater  role 
in  helping  technologies  reach  high  levels  of  maturity  before  product 
development  begins.  For  example,  given  that  a  technology  has  sufficient 
potential  for  application  to  a  weapon  system,  the  GAO  suggested  that,  at  a 
minimum,  an  S&T  organization  should  be  responsible  for  taking  a  technology  to 
TRL  6  before  it  is  handed  off  to  a  program  office  at  the  program  definition 
and  risk  reduction  phase.  Further,  the  GAO  indicated  that  during  this  phase, 
the  weapon  system  program  manager  would  be  responsible  for  maturing  the 
technology  to  TRL  7  before  the  technology  can  be  included  in  engineering  and 
manufacturing  development  programs.  The  GAO  asserted  that,  in  a  situation 
where  a  single,  design-pacing  technology  is  to  be  developed  for  a  known 
application-like  the  non-penetrating  periscope-an  S&T  organization  should  be 
required  to  mature  the  technology  to  TRL  7  before  it  is  turned  over  to  a 
product  development  manager.  The  GAO  also  asserted  that  S&T  organization 
could  play  a  similar  role  when  a  significant  new  technology  is  being  readied 
for  insertion  into  an  existing  weapon  system.  Finally,  when  multiple  new 
technologies  are  to  be  brought  together  to  create  a  weapon  system,  the  GAO 
indicated  that  S&T  organizations  should  be  required  to  bring  key  technologies 
to  TRL  6  and  then  become  part  of  a  hybrid  organization  with  product  developers 
to  integrate  the  technologies  and  bring  them  to  TRL  7  before  handing  full 
responsibilities  off  to  a  product  develop  manager. 

DOD  RESPONSE:  Concur.  The  DoD  S&T  organizations  should  be  involved  in  „ 
maturing  technologies  to  high  levels  of  maturity  like  TRL  6  before 
transitioning  to  EMD.  The  Department  concurs  with  the  GAO  that  the  weapon 
system  program  manager  should  assure  that  technology  is  matured  to  TRL  7 
before  insertion  into  a  new  system.  (Maturing  a  technology  to  Technology 
Readiness  Level  6,  however,  may  be  sufficient  for  the  purposes  of  a  preplanned 
product  improvement  program  or  block  upgrade  program  for  a  system  which  has 
already  been  deployed.)  DoD  also  concurs  that  the  relevant  S&T  organizations 
must  continue  to  support  system  managers  in  all  phases  of  development.  This 
is  done  in  the  DOD  today,  typically  through  the  Integrated  Product  Development 
Process.  Integrated  Product  Teams  are  employed  in  the  Services  today  for  life- 
cycle  system  develbpment  and  acquisition. 
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RECOMMENDATION  5 ;  To  help  guard  against  the  possibility  that  the  more  basic 
research  and  technology  development  activities  would  be  compromised  by  having 
S&T  organizations  routinely  take  key  technologies  to  TRL  6  or  higher,  the  GAO 
recommended  that  the  Secretary  of  Defense  extract  lessons  learned  from  the 
M  7  ,  fir  non-penetrating  periscope,  the  Advanced  Amphibious  Assault  Vehicle  (AAAV) 

[NOW  On  pp.  /  ana  DO.  program,  the  Army's  Future  Scout  Program,  and  other  Advanced  Technology 

Demonstration  (ATD)  programs  and  Advanced  Concept  and  Technology  Demonstration 
(ACTD)  programs.  Specifically,  the  GAO  asserted  that  the  Secretary  should 
assess  whether  the  resources  needed  in  these  efforts  to  enable  S&T 
organizations  to  play  a  leading  role  in  the  development  of  technologies  and, 
in  some  cases,  preliminary  system  design,  detracted  from  or  displaced  more 
basic  research  and  technology  developments  efforts. 

POD  RESPONSE;  Concur.  The  recommendation  that  the  Department  assess  the 
impact  upon  other  S&T  resources  is  appropriate. 

RECOMMENDATION  6 :  The  GAO  further  recommended  that  the  Secretary  of  Defense 
empower  managers  of  product  development  programs  to  refuse  to  accept  key 
technologies  with  low  levels  of  demonstrated  maturity.  The  GAO  indicated  that 
NOW  on  pp. 7  and  65.  the  Secretary  can  encourage  this  behavior  through  supportive  decisions  on 

individual  programs,  such  as  by  denying  proposals  to  defer  the  development  of 
key  technologies  and  by  favoring  proposals  to  lengthen  schedules  or  lessen 
requirements  to  reduce  technological  risk  early. 

POD  RESPONSE:  Concur.  Program  managers  are  responsible  for  balancing  the 
many  different  requirements  for  a  system,  including  cost,  schedule  and 
performance.  In  their  on-going  assessment  and  management  of  programs, 
managers  continually  re-evaluate  the  maturity,  projected  schedule  and  risk 
associated  with  relevant  technologies.  Decisions  by  the  acquisition  executive 
are  made  with  input  from  the  program  managers.  Thus  program  managers  already 
have  the  ability  to  reject  inappropriately  mature  technologies,  and,  to  the 
extent  technology  immaturity  impacts  acquisition  baselines,  to  advise  the 
acquisition  executive  of  feasible  alternatives  and  make  recommendations. 
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